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Abstract 


Objectives:  A  pilot  study  was  conducted  to  investigate  the  effectiveness  of  continuous  haptic 
peripheral  displays  in  supporting  multiple  UAV  supervisory  control.  Background:  Previous 
research  shows  that  continuous  auditory  peripheral  displays  can  enhance  operator  performance  in 
monitoring  events  that  are  continuous  in  nature,  such  as  monitoring  how  well  UAVs  stay  on  their 
pre-planned  courses.  This  research  also  shows  that  auditory  alerts  can  be  masked  by  other 
auditory  information.  Command  and  control  operations  are  generally  performed  in  noisy 
environments  with  multiple  auditory  alerts  presented  to  the  operators.  In  order  to  avoid  this 
masking  problem,  another  potentially  useful  sensory  channel  for  providing  redundant 
information  to  UAV  operators  is  the  haptic  channel.  Method:  A  pilot  experiment  was  conducted 
with  13  participants,  using  a  simulated  multiple  UAV  supervisory  control  task.  All  participants 
completed  two  haptic  feedback  conditions  (continuous  and  threshold),  where  they  received  alerts 
based  on  UAV  course  deviations  and  late  arrivals  to  targets.  Results :  Threshold  haptic  feedback 
was  found  to  be  more  effective  for  late  target  arrivals,  whereas  continuous  haptic  feedback 
resulted  in  faster  reactions  to  course  deviations.  Conclusions:  Continuous  haptic  feedback 
appears  to  be  more  appropriate  for  monitoring  events  that  are  continuous  in  nature  (i.e.,  how  well 
a  UAV  keeps  its  course).  In  contrast,  threshold  haptic  feedback  appears  to  better  support 
response  to  discrete  events  (i.e.,  late  target  arrivals).  Future  research:  Because  this  is  a  pilot 
study,  more  research  is  needed  to  validate  these  preliminary  findings.  A  direct  comparison 
between  auditory  and  haptic  feedback  is  also  needed  to  provide  better  insights  into  the  potential 
benefits  of  multi-modal  peripheral  displays  in  command  and  control  of  multiple  UAVs. 
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Introduction 


In  current  military  conflicts,  the  unmanned  aerial  vehicle  (UAV)  is  one  of  the  most  highly  sought 
after  air  assets  (Whitney,  2007).  UAVs  provide  real  time  information  and  support  to  battlefield 
operations  remotely  from  either  local  or  global  control  station  platforms.  Each  of  these 
operational  environments  presents  fluid  atmospheres  of  noise  and  activity. 

Currently  many  medium  and  high  altitude  UAY  operations  require  multiple  operators  to  control 
a  single  UAV;  the  desire  for  the  future  is  to  invert  the  numbers  and  have  a  single  operator  control 
multiple  UAVs  (Culbertson,  2006).  To  achieve  this,  research  is  needed  to  understand  how  best 
to  aid  and  integrate  the  human  operator  into  a  system  of  semi-autonomous  UAVs.  Due  to 
potential  information  overload  of  operators  in  visually-saturating  domains  such  as  UAV 
supervisory  control,  reducing  mental  workload  by  offloading  some  cognitive  processing  to  the 
other  senses,  primarily  auditory  and  haptic,  is  critical. 

Graham  &  Cummings  (2007)  investigated  whether  continuous  auditory  displays  could  be  applied 
to  a  command  and  control  setting,  particularly  for  supervisory  control  of  one  or  more  unmanned 
vehicles  by  a  single  operator  (see  the  Appendix).  Specifically,  two  continuous  audio  displays 
were  mapped  to  the  alerting  of  two  human  supervisory  task  problems:  UAV  course  deviations 
and  late  target  arrivals.  The  findings  suggest  that  for  UAV  course  conformance,  the  continuous 
audio  alert  resulted  in  better  performance  than  discrete  audio,  regardless  of  the  alert  type 
provided  for  late  target  arrivals  (i.e.,  continuous  or  threshold).  For  late  arrivals,  continuous  late 
arrival  alert  in  general,  and  the  threshold  late  arrival  alert  only  when  accompanied  by  threshold 
course  deviation  alert  resulted  in  good  performance.  When  accompanied  by  continuous  course 
deviation  alert,  the  threshold  late  arrival  alert  generated  the  worst  performance.  The  authors 
attributed  this  performance  degradation  to  a  possible  masking  problem  where  the  continuous 
course  deviation  audio  alert  saturated  the  audio  channel  to  a  point  that  the  late  arrival’s  threshold 
beep  alerts  were  not  heard,  resulting  in  longer  late  arrival  reaction  times. 

Auditory  feedback  could  enhance  operator  performance,  but  command  and  control  settings  are 
generally  noisy,  and  auditory  alerts  may  be  masked,  as  was  highlighted  in  the  previous  study. 
Another  sensory  channel  that  can  be  utilized  to  provide  feedback  to  operators  is  the  haptic 
channel.  Haptic  feedback  has  been  shown  to  support  operator  performance  in  different  domains, 
such  as  driving  (Schumann,  Godthelp,  Farber,  &  Wontorra,  1993;  Suzuki  &  Jansson,  2003),  and 
aviation  (Sklar  &  Sarter,  1999).  Sklar  &  Sarter  (1999)  showed  that  when  compared  to  visual 
cues,  both  tactile,  and  redundant  visual  and  tactile  cues  result  in  faster  response  to  and  higher 
detections  rates  for  mode  transitions  in  an  automated  cockpit  system.  Burke  et  al.  (2006) 
compared  the  effects  of  visual-auditory  and  visual-tactic  feedback  using  a  meta-analytic 
approach  on  43  different  studies.  Adding  an  additional  modality  was  found  to  enhance 
performance  overall,  with  visual-auditory  feedback  most  effective  for  moderate  workload,  single 
task  conditions,  and  visual-tactile  feedback  more  effective  for  high  workload,  multiple  task 
conditions.  As  the  command  and  control  tasks  shift  from  many  operators  controlling  one  vehicle 
to  one  controlling  many,  high  operator  workload  can  become  a  major  issue.  Moreover,  as 
mentioned  above,  auditory  warnings  provided  to  the  operator  can  be  masked  by  environmental 
noise  and  other  auditory  alerts.  Therefore,  haptic  feedback  may  provide  better  support  for 
command  and  control,  and  merits  research  in  this  domain. 
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As  a  follow  on  to  the  research  of  Graham  &  Cummings  (2007),  a  pilot  study  was  conducted  to 
assess  the  potential  benefits  of  continuous  haptic  feedback  in  UAV  supervisory  control.  This 
report  presents  the  findings  of  this  study  which  in  particular  compared  continuous  haptic 
feedback  to  discrete  haptic  feedback. 


Experimental  Protocol 


Objective 

The  objective  of  this  experiment  was  to  assess  and  compare  the  effects  of  continuous  and 
discrete  haptic  feedback  in  supporting  multiple  UAV  supervisory  control.  Participants  were 
presented  with  continuous  or  discrete  haptic  feedback  on  UAV  course  deviations  and  late  target 
arrivals  while  performing  an  auditory  secondary  task. 

Participants 

Thirteen  participants  completed  the  study.  The  majority  of  participants  were  Humans  and 
Automation  Laboratory  members.  The  experiment  took  between  2.5  and  3  hours  to  complete. 
The  threshold  condition  data  for  the  first  participant  could  not  be  included  in  the  analysis  due  to 
a  bug  in  the  experimental  software  which  was  later  fixed.  Another  participant’s  data  was 
completely  dropped  from  analysis  because  of  the  test  proctor  failing  to  administer  proper 
training.  This  participant  was  also  an  outlier  in  regards  to  their  age  (i.e.,  63).  The  remaining 
participants  consisted  of  five  females  and  seven  males,  ages  ranging  from  18  to  45  years  (mean  = 
27,  standard  deviation  =  8.14). 

Apparatus 

The  simulated  task  of  controlling  multiple  UAVs  was  carried  out  in  Multiple  Autonomous 
Unmanned  Vehicle  Experimental  (MAUVE)  test  bed  developed  by  The  Massachusetts  Institute 
of  Technology’s  Humans  and  Automation  Laboratory  (HAL).  MAUVE  is  a  two  screen  interface 
that  allows  the  simulation  of  any  number  of  unmanned  vehicles  conducting  strike  operations  in 
hostile  environments  (Figure  1).  MAUVE,  for  this  experiment,  allowed  the  operator  to  arm  and 
fire  on  targets  with  various  priorities  while  monitoring  each  UAV’s  flight  of  path,  event  timeline, 
and  radio  traffic  (The  Appendix  provides  more  details). 

The  test  bed  provided  a  map  for  geo-spatial  tracking  of  UAVs  on  their  preset  flight  paths  along 
with  an  interaction  control  panel  for  making  UAV  control  inputs  (Map  Display,  Figure  1). 
Control  inputs  included  arming  and  firing  a  UAV  to  destroy  targets,  directing  specific  path 
changes,  and  acknowledging  radio  calls.  The  path  changes  included  commanding  a  UAV  to  skip 
a  target  to  get  a  UAV  back  on  schedule.  The  operator  could  also  reset  the  UAV’s  onboard 
navigation  to  get  the  vehicle  back  on  course  if  it  drifted. 

A  timeline  for  each  UAV,  a  scheduling  decision  support  tool  for  each  UAV,  and  chat  interfaces 
were  also  provided  (Timeline  Display,  Figure  1).  The  timeline  display  helped  operators  ensure 
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UAVs  would  be  on  time  to  targets  by  monitoring  when  they  would  arrive  at  targets,  and  in  what 
phase  they  were  currently  located. 


Map  Display 


Timeline  Display 


Figure  1:  Multiple  Autonomous  Unmanned  Vehicle  Experimental  (MAUVE)  test  bed. 


Figure  2:  The  Multi-Modal  Workstation  (MMWS). 


The  MAUVE  simulation  environment  was  run  on  a  multi-modal  workstation  (MMWS).  Pictured 
in  Figure  2,  the  MMWS  is  a  four  screen  computer  display  (Osga,  Van  Orden,  Campbell, 
Kellmeyer,  &  Lulue,  2002).  The  three  screens  across  the  top  were  21  inches  and  set  to  display  at 
1280  x  1024  pixel,  16  bit  color  resolution,  and  the  15  inch  bottom  center  screen  was  set  at  1024  x 
768  pixels,  32  bit  color  resolution.  The  computer  used  to  run  the  simulator  was  a  Dell  Optiplex 
GX280  with  a  Pentium  4  processor  and  an  Appian  Jeronimo  Pro  4-Port  graphics  card. 
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Participants  controlled  the  simulator  through  a  generic  corded  computer  mouse.  Throughout  the 
experiment,  the  top  left  display  showed  the  mission  objectives  and  priorities,  and  the  bottom 
center  displayed  the  color-coding  for  MAUVE.  The  top  center  display  contained  the  left 
MAUVE  map  and  interaction  display,  and  the  top  right  display  included  the  right  MAUVE 
timeline  and  decision  support  display.  During  the  test  sessions,  the  screen  recordings  for  the  left 
and  right  MAUVE  displays  were  captured  by  Camtasia  ®  recording  software. 

An  over-the-head  headset  with  sealed  around-the-ear  cups  (Sensimetrics  HDiSP,  Figure  3)  was 
used  to  present  the  auditory  secondary  task  used  in  the  experiment.  This  headset  provides 
ambient  noise  attenuation  ranging  from  35  to  40  db  for  the  spectrum  between  500  Hz  and  8  kHz. 


Figure  3:  Sensimetrics  HDiSP. 


Haptic  feedback  was  presented  through  a  pressure  vest  that  expanded  based  on  late  target 
arrivals,  and  a  wristband  that  vibrated  to  indicate  course  deviations  (Figure  4,  Figure  5).  The 
hardware  for  the  pressure  vest  included  miniature  face-mounted  control  valves  and  eight  place 
manifolds  from  the  Lee  Inc.  “LH  Series”.  The  bladders  inflated  to  20  psi.  The  wristband  used 
five  Electronics  Inc.  “QX”  miniature  vibration  motors.  The  frequency  of  the  vibrations 
depended  on  the  tightness  with  which  the  motors  were  held  against  the  body,  but  the  nominal 
frequency  was  100  Hz.  Participants  wore  the  wristband  on  their  right  hand  which  they  also 
controlled  the  mouse  with.  How  these  haptic  devices  behaved  for  late  target  arrivals,  and  course 
deviations  will  be  described  in  further  detail  in  a  subsequent  section. 
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Figure  4: Haptic  feedback  hardware:  pressure  vest,  and  vibrating  wristband. 


Figure  5.  Vibrating  wristband  and  air  pressure  bladders  for  the  vest. 


Experimental  Tasks 

The  task  used  in  this  experiment  was  identical  to  the  multiple  UAV  level  scenario  tested  by 
Graham  &  Cummings  (2007),  where  participants  had  to  control  four  UAVs.  In  particular, 
participants  had  to  respond  to  UAV  course  deviations  and  late  target  arrivals  while  performing 
an  auditory  secondary  task.  The  primary  task  of  the  operator  was  to  ensure  weapons  were 
dropped  on  the  correct  targets  during  the  specified  windows  of  opportunity.  However, 
unexpected  head  winds  or  crosswinds  could  cause  UAVs  to  slow  their  speed  or  drift  off  course, 
requiring  the  operator  to  take  corrective  action  for  projected  target  late  arrivals  or  course 
deviations.  Late  arrivals  were  a  higher  priority  than  course  deviations.  In  addressing  late  arrivals, 
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the  priority  of  the  target  (high,  medium,  or  low)  also  had  to  be  addressed,  such  that  it  was  more 
important  to  hit  a  high  priority  target  than  a  medium  priority  target,  and  so  on.  In  addition, 
operators  also  had  to  ensure  the  UAYs  would  return  to  base  by  a  specified  time. 

Course  deviations:  The  green  line  in  Figure  6  indicates  the  plotted  and  desired  flight  path  of  a 
UAV.  Anytime  the  UAV  has  completely  departed  from  the  green  course  line,  it  is  considered  to 
have  deviated.  During  the  simulation,  the  UAVs  constantly  fluctuated  back  and  forth  on  their 
flight  paths,  but  participants  were  trained  to  respond  when  the  UAV  actually  was  no  longer 
touching  the  course  line.  Figure  6  illustrates  UAV  3  on  course,  almost  off  course,  and  off  course. 
The  instructions  on  how  to  comply  with  recovery  rules  for  course  deviations  were  to  click  “Reset 
Navigation.”  This  theoretically  reset  the  onboard  navigation  in  the  UAV  and  caused  it  to  fly  back 
to  the  planned  flight  path. 
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Figure  6:  A  course  deviation  event. 

Late  target  arrivals:  Late  arrivals  were  caused  when  a  UAV  slowed  down  because  of  unforeseen 
headwinds  and  was  no  longer  able  to  reach  a  target  in  time  to  complete  the  firing  mission, 
according  to  its  originally  scheduled  time.  Figure  7  shows  the  visual  representation  for  a  late 
arrival.  A  black  rectangle  on  the  timeline  shows  when  the  UAV  is  going  to  arrive  at  the  labeled 
target,  and  changes  from  black  to  red  when  the  UAV  is  late.  This  can  also  be  seen  when  the 
black  rectangle  moves  past  the  orange  window  of  opportunity.  On  the  timeline  below,  the  red 
rectangle  labeled  “T-1L”  indicates  that  the  UAV  is  going  to  be  late  to  target  T-1L.  In  contrast, 
the  black  rectangle  labeled  “T-2M”  indicates  that  the  UAV  is  going  to  be  on  time  to  target  T-2M. 
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Figure  7:  A  late  arrival  event. 

Participants  were  instructed  to  skip  the  low  priority  targets  by  clicking  the  “Skip  Target”  button, 
which  caused  the  UAV  to  proceed  immediately  to  the  next  target.  For  medium  and  high  priority 
targets,  the  option  was  given  to  use  a  decision  support  visualization  tool  (DSV),  which  illustrated 
the  effect  of  requesting  delay  for  the  current  late  target  on  the  remaining  targets  for  that  UAV’s 
flight  plan.  With  the  assistance  of  the  information  presented  in  the  DSV,  participants  then 
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decided  whether  they  should  click  the  “Skip  Target”  button  or  “Request  Delay”  button.  If  they 
clicked  “Request  Delay,”  they  then  had  to  wait  to  see  if  a  delay  was  granted,  and  if  the  delay  was 
granted  the  UAV  was  then  no  longer  late  to  the  target  and  would  be  able  to  destroy  it.  For  a  high 
priority  target  the  participants  were  instructed  to  employ  DSV  before  requesting  delay  or 
skipping  target. 

The  instructions  on  how  to  comply  with  recovery  rules  for  target  late  arrivals  were: 

•  Low  Priority  -  Click  “Skip  Target” 

•  Medium  Priority  -  Click  “Skip  Target”  or  Employ  DSV. 

•  High  Priority  -  Employ  DSV  before  Requesting  Delay  or  Clicking  “Skip  Target” 

Auditory  secondary  task:  As  a  secondary  workload  task,  participants  were  instructed  to  monitor  a 
recording  of  continual  air  traffic  radio  chatter  for  the  word  “Push.”  The  word  “Push”  occurred 
62  times  in  a  30  minute  session,  with  an  average  of  27  seconds  between  each  “Push”  radio  call. 
To  acknowledge  the  radio  call,  participants  clicked  an  “Acknowledge  Push”  button  on  the 
display. 

Task  priorities:  The  operator  was  given  the  following  as  the  primary  objective  for  each  mission: 

Make  sure  the  UAV(s)  maximize  the  number  of  targets  engaged  as  well  as  arrive  back  at 
the  base  safely. 

Further,  supervision  of  each  of  the  UAV(s)  for  each  mission  was  broken  down  into  the  following 
prioritized  sub-tasks,  from  highest  priority  to  lowest: 

1.  Return  to  base  (RTB)  within  the  time  limit  for  the  mission  (this  limit  was  clearly 
marked). 

2.  Comply  with  recovery  rules  for  course  deviations. 

3.  Comply  with  recovery  rules  for  target  late  arrivals. 

4.  Destroy  all  targets  before  their  time  on  target  (TOT)  window  ended. 

5.  Avoid  taking  damage  from  enemies  by  avoiding  all  threat  areas. 

6.  Acknowledge  all  “Push”  radio  calls. 

Participants  were  trained  to  follow  these  priorities,  and  the  objectives  and  priorities  above  were 
displayed  to  them  throughout  the  experiment.  Within  the  priorities,  recovery  rules  of  how  to 
recover  the  UAV  from  a  course  deviation  or  late  arrival  were  also  displayed. 

Haptic  Feedback 

Continuous  feedback  for  late  target  arrivals:  The  pressure  vest  inflated  with  the  number  of  air 
bladders  filled  proportional  to  the  target  priority,  and  stayed  on  until  the  participant  responded  to 
the  late  arrival.  Each  air  bladder  inflated  to  20  psi.  There  were  two  sets  of  air  bladders,  with  a 
total  of  eight  bladders  (Figure  5).  Each  set  contained  four  bladders.  Four  (top  most  two  in  each 
set),  six  (top  most  three  in  each  set),  and  eight  (all)  bladders  were  filled  for  low,  medium,  and 
high  priority  targets,  respectively.  The  bladders  stayed  inflated  until  the  operator  addressed  the 
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issue  with  corrective  action  or  the  issue  disappeared  because  the  UAV  has  continued  to  the  next 
target. 

Threshold  feedback  for  late  target  arrivals:  The  pressure  vest  inflated  for  late  arrivals  and  stayed 
on  for  2000  ms  including  the  time  it  took  to  pump  up  the  bladders  to  20  psi.  All  eight  bladders 
were  filled  for  2000  ms,  independent  of  target  priority. 

Continuous  feedback  for  course  deviations:  The  wristband  buzzed  continually  throughout  the 
experiment  and  the  buzzing  intensified  as  the  UAV  deviated  further  off  course.  The  motors 
always  buzzed  at  their  nominal  100  Hz  frequency,  and  were  always  turned  on  for  300ms  for  a 
given  activation.  As  course  deviation  increased,  the  number  of  activated  motors  increased  and 
the  amount  of  time  that  passed  between  each  activation  (i.e.,  gap  time)  was  decreased.  The 
number  of  motors  turned  on  and  the  duration  between  activations  were  dynamically  mapped  to 
the  course  deviation  parameter  defined  in  MAUVE  as  degrees  off  the  course  (Figure  8).  For 
example,  at  a  very  low  course  deviation  (e.g.,  “3  degrees”  in  MAUVE),  a  single  motor  would 
buzz  (at  100  Hz)  for  300  ms  approximately  every  6  seconds.  When  the  deviation  increased  to 
nine  degrees,  three  motors  were  activated,  and  the  length  of  time  between  activations  was  2.5 
seconds. 
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Figure  8.  Number  of  motors  activated  and  gap  time  between  each  activation  as  a  function  of  UAV  course 

deviation. 

Threshold  feedback  for  course  deviations:  When  a  UAV  deviated  from  its  course  by  10  degrees, 
all  five  motors  on  the  wristband  turned  on  for  600  ms  at  their  nominal  frequency  of  100  Hz. 
Falling  below  the  threshold  and  rising  past  it  again  caused  another  activation  of  the  motors. 

Experimental  Design 

The  independent  variable  of  interest  was  haptic  feedback  type,  which  had  two  levels:  continuous 
(continuous  course  deviation  and  continuous  late  arrival),  and  threshold  (threshold  course 
deviation  and  threshold  late  arrival).  The  experiment  was  a  single  factor  repeated  measures 
design.  That  is,  each  participant  completed  both  the  continuous  and  the  threshold  feedback 
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conditions.  The  order  of  presentation  for  the  two  conditions  was  counterbalanced  and  randomly 
assigned  to  participants. 


While  the  experiment  conducted  by  Graham  &  Cummings  (2007)  included  a  factor  that  varied 
number  of  UAVs  under  control  (from  1  to  4),  the  current  experiment  focused  on  the  4  vehicle 
condition,  which  represents  the  higher  workload  situation. 

Dependent  Variables 

Dependent  variables  included  the  number  of  missed  course  deviations,  reaction  time  to  correct 
course  deviations,  number  of  missed  late  arrivals,  reaction  time  to  correct  projected  late  arrivals, 
NASA  TLX  scores,  and  the  number  of  acknowledged  radio  calls. 

In  each  test  scenario,  participants  were  expected  to  respond  to  four  triggered  course  deviations 
and  four  late  target  arrivals.  Anytime  the  participant  failed  to  respond  to  one  of  four  triggered 
course  deviations,  an  error  of  omission  was  counted.  Similarly,  errors  of  omission  for  late 
arrivals  occurred  when  the  participant  failed  to  respond  to  one  of  the  four  triggered  late  arrivals. 
Primary  task  performance  was  assessed  using  errors  of  omissions  for  and  reaction  times  to 
course  deviations  and  late  target  arrivals. 

The  number  of  push  call  responses  was  analyzed  to  assess  participants’  secondary  task 
engagement.  Subjective  workload  was  assessed  using  NASA  Task  Load  Index.  The  NASA  Task 
Load  Index  gathered  participants’  subjective  assessment  on  a  scale  of  1  to  20  of  mental  demand, 
physical  demand,  temporal  demand,  effort,  performance,  and  frustration.  A  participant  was  then 
asked  to  rate  these  six  dimensions  against  each  other  to  determine  their  importance  in  the 
participant’s  workload  (Hart  &  Staveland,  1988).  Since  there  was  no  physical  demand  in  this 
experiment,  each  participant  was  told  to  rate  the  physical  demand  so  that  it  was  zeroed  out  of  the 
score. 


Procedure 

Each  participant  experienced  three  separate  consecutive  phases  in  completing  the  experiment. 
Each  had  a  60  to  70  minute  training  session  followed  by  a  70  to  80  minute  test  session,  and  then 
a  10  minute  post-test  survey.  For  the  training  session,  each  participant  completed  a  demographics 
survey,  received  standardized  training  from  a  PowerPoint®  tutorial,  and  a  thirty  minute  practice 
test  session  with  a  trial  run  through  of  NASA  TLX.  If  required,  a  participant  could  do  an 
additional  10  minute  practice  session  to  ensure  understanding  of  the  proper  MAUVE  control 
actions. 

Before  each  feedback  condition,  the  participants  were  presented  with  a  demonstration  of  how  the 
specific  feedback  worked,  and  hence  were  exposed  to  all  the  haptic  feedback  they  would  feel  in  a 
test  session.  They  were  presented  with  their  respective  late  arrival  and  course  deviation  alerts,  as 
well  as  the  radio  chatter. 

After  the  training  was  completed,  each  participant  then  completed  both  the  continuous  and 
threshold  feedback  conditions  in  two  separate  test  session.  At  the  conclusion  of  every  test 
session,  MAUVE  generated  a  data  log,  with  all  the  previously  discussed  dependent  variables. 
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Additionally,  participants  completed  a  NASA  TLX  following  each  test  session.  At  the 
conclusion  of  the  test  session,  participants  completed  a  post-test  survey,  and  were  thanked  for 
their  time  and  involvement. 


Results 

For  statistical  analysis,  a  repeated  measures  ANOVA  was  conducted  on  dependent  variables. 
Normality  and  homogeneity  of  variances  assumptions  were  checked.  As  mentioned  above,  there 
were  four  course  deviations  and  four  late  arrivals  in  each  condition.  The  average  reaction  times 
were  analyzed  for  both  course  deviations  and  late  arrivals. 


Course  Deviations 

Out  of  the  92  course  deviations  presented  to  the  12  participants,  only  1  was  missed.  This  miss 
was  in  the  threshold  condition.  Figure  9  shows  the  means  for  the  course  deviation  reaction  times 
for  the  two  haptic  feedback  conditions.  Continuous  haptic  feedback  resulted  in  an  estimated  2.46 
sec  faster  course  deviation  reaction  time  than  threshold  feedback  (t( 21)  =  -4.1,  p  =  0.0005,  95% 
Cl  =  -3.70, -1.21). 
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Figure  9:  Course  deviation  reaction  times  (means  and  standard  error  bars). 


Late  Arrivals 

All  participants  responded  to  all  of  the  4  late  arrivals  presented,  thus  there  were  no  late  arrival 
errors  of  omission. 

For  late  arrival  reaction  times,  out  of  the  92  late  arrival  data  points,  five  outliers  had  to  be 
removed  from  the  analysis,  which  were  more  than  three  standard  deviations  away  from  the  mean. 
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These  values  were  between  34  and  42  seconds,  which  indicated  that  the  participants’  actions 
were  not  in  response  to  initiation  of  feedback.  Four  of  these  outliers  were  in  threshold  feedback 
condition  and  one  was  in  continuous  feedback  condition.  Continuous  feedback  was  supposed  to 
stay  on  until  the  participant  responded  to  the  late  arrival.  The  outlier  response  during  the 
continuous  feedback  and  the  two  of  the  outlier  responses  during  threshold  feedback  belonged  to 
one  participant.  This  participant  indicated  that  he  could  not  feel  the  inflation  of  the  pressure  vest 
very  well  in  general,  and  also  noted  that  the  vest  did  not  inflate  for  the  first  late  target  arrival  in 
the  threshold  condition  (this  corresponded  to  one  of  the  outlier  responses).  The  two  other  outlier 
responses  observed  during  the  threshold  condition  belonged  to  two  different  participants.  For  one 
of  these  participants,  the  experimenter  observed  that  the  vest  took  longer  time  inflating/deflating 
than  normal.  The  cause  of  the  other  outlier  response  is  unclear,  but  may  also  be  a  vest 
malfunction. 

Threshold  haptic  feedback  resulted  in  an  estimated  1.74  sec  faster  late  arrival  reaction  time  than 
continuous  feedback  (t( 21)  =  -2.51,  p  =  0.02,  95%  Cl  =  -3.18,  -0.30).  Figure  10  shows  the  means 
for  the  late  arrival  reaction  times  for  the  two  haptic  feedback  conditions. 


Figure  10:  Late  arrival  reaction  times  (means  and  standard  error  bars). 


Secondary  Task  Engagement  and  Subjective  Workload 

The  difference  between  continuous  and  threshold  feedback  types  was  not  significant  for  the 
number  of  push  call  responses  (t(18)  =  1.98,  p  =  0.22),  as  well  as  for  NASA  TLX  scores  (/(2 1 )  = 
0.26,  p  =  0.8)  (Figure  11). 
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Figure  11:  Number  of  push  call  responses  and  subjective  workload  (means  and  standard  error  bars). 


Post-test  Feedback 

Almost  all  participants  thought  that  the  haptic  cues  helped  them  in  responding  to  both  late 
arrivals  as  well  as  course  deviations.  Some  even  said  that  they  depended  on  haptic  as  the  primary 
source  of  identifying  an  off-nominal  state.  These  participants  waited  for  the  haptic  feedback  and 
then  confirmed  the  need  for  an  action  visually.  The  haptic  feedback  was  perceived  to  be  useful, 
especially  for  course  deviations  since  the  farthest  off-course  UAV  was  always  highlighted  in  red 
and  visual  information  alone  was  not  enough  to  determine  if  the  course  deviation  required  action. 
Despite  the  general  positive  attitude  towards  haptic  feedback,  there  were  also  concerns  about 
operator  annoyance  with  prolonged  use  of  these  haptic  feedback  devices. 

As  for  the  pressure  vest,  the  majority  of  participants  thought  that  the  differential  inflation  based 
on  target  priority  was  not  as  informative  as  simply  the  inflation  of  the  vest.  Most  said  that 
feedback  was  very  salient  but  the  differences  between  target  priorities  were  indistinguishable. 
Participants,  in  general,  suggested  keeping  the  vest  inflated  until  the  late  arrival  was  addressed. 
However,  one  participant  also  suggested  adding  pulses  to  the  inflated  vest,  since  they  felt  that  if 
they  did  not  attend  to  the  late  arrival  as  soon  as  they  got  feedback,  they  may  forget  to  do  so. 
There  were  two  participants  who  found  the  pressure  vest  to  be  cumbersome.  One  of  these 
participants  indicated  that  he  did  not  feel  the  inflation  as  much  as  he  noticed  vibrations  on  the 
wristband. 

Most  participants  preferred  the  threshold  feedback  for  the  wristband.  Participants  in  general  did 
not  like  the  continual  buzzing;  some  even  felt  that  they  responded  slower  with  this  type  of 
feedback.  However,  some  still  thought  that  the  change  in  the  intensity  of  buzzing  did  help 
indicate  how  far  off  course  they  were.  A  suggestion  received  from  multiple  participants  was  to 
design  the  wristband  to  not  continually  buzz,  but  still  have  a  gradient  buzz  after  a  threshold 
course  deviation  point  was  exceeded.  As  mentioned  before,  all  participants  wore  the  wristband 
on  their  right  hands.  One  participant  suggested  wearing  the  wristband  on  their  non-dominant 
hand. 


13 


Discussion 


A  pilot  experiment  was  conducted  to  evaluate  continuous  peripheral  haptic  feedback  for 
supporting  multiple  UAV  supervisory  control.  The  results  of  this  experiment  showed  that 
continuous  haptic  feedback  promoted  better  performance  for  course  deviations,  whereas 
threshold  haptic  feedback  resulted  in  better  performance  for  late  target  arrivals.  In  general, 
workload  and  secondary  task  engagement  did  not  differ  between  the  two  feedback  types. 

This  pilot  study  suggests  that  continuous  haptic  feedback  can  be  a  promising  way  to  help  single 
operators  supervise  multiple  semi-autonomous  vehicles.  The  preliminary  results  indicate  that  a 
threshold  haptic  alert  is  more  effective  for  tasks  that  require  monitoring  discrete  events  (e.g.,  late 
arrival  to  a  target),  whereas  a  continual  alert  is  more  appropriate  for  tasks  that  require  monitoring 
events  that  are  continuous  in  nature  (e.g.,  monitor  how  well  a  UAV  keeps  its  course).  Graham  & 
Cummings  (2007)  also  found  that  continuous  auditory  feedback  resulted  in  faster  response  to 
course  deviations  when  compared  to  discrete  auditory  feedback.  They  also  found  that  discrete 
feedback  resulted  in  slowest  response  to  late  target  arrivals  when  accompanied  with  continuous 
feedback  on  course  deviations,  which  was  attributed  to  a  possible  masking  effect. 

In  this  study,  continuous  course  deviation  alert  was  presented  with  continuous  late  arrival  alert, 
and  threshold  course  deviation  alert  was  presented  with  threshold  late  arrival  alert.  It  is  unclear 
how  continuous  and  discrete  haptic  feedback  would  interact  in  supporting  operator  response.  For 
example,  a  combination  of  continuous  course  deviation  and  threshold  late  arrival  alerts  can  free 
more  perceptual/mental  resources,  and/or  enable  the  operator  to  better  timeshare  between  the  two 
supervisory  tasks  by  supporting  task  switching.  The  current  study  was  not  designed  to  assess  the 
interacting  effects  of  continuous  and  discrete  haptic  feedback,  and  further  research  is  needed  to 
understand  how  different  feedback  types  interact  in  aiding  supervisory  control.  Moreover, 
Graham  &  Cummings  (2007)  used  only  one  device  delivering  feedback  (i.e.,  headphones), 
whereas  in  the  current  experiment,  there  was  a  separate  feedback  device  for  the  two  different 
monitoring  tasks  (i.e.,  wristband  for  course  deviations,  and  pressure  vest  for  late  arrivals). 
Therefore,  the  type  of  device  which  provides  feedback  may  also  influence  results.  These 
feedback  types  should  be  tested  in  more  detail  to  detangle  the  effects  of  feedback  type  from  the 
effects  of  feedback  device.  In  addition,  further  testing  with  a  combination  of  haptic  and  audio 
alerts  is  warranted  to  determine  the  best  alerting  architecture. 

An  additional  interesting  finding  in  this  study  was  the  mismatch  between  participants’  subjective 
responses  and  actual  performance  for  course  deviations.  Participants  in  general  liked  the 
threshold  feedback  for  course  deviations,  and  some  even  said  that  continuous  feedback 
undermined  their  performance.  Even  if  the  continual  buzzing  of  the  wristband  was  annoying  to 
the  participants,  they  still  performed  better  with  continuous  feedback.  It  is  unclear  how 
operators’  acceptance  of  these  alerts  would  change  performance  with  long  term  use.  Acceptance 
of  an  alert  has  an  important  influence  on  the  use  and  hence  the  effectiveness  of  it  (Donmez, 
Boyle,  Lee,  &  McGehee,  2006),  but  effectiveness  may  also  guide  perceived  usefulness  and 
therefore  acceptance.  For  example,  low  acceptance  observed  in  this  study  may  lead  to  disuse  of 
the  alerts  in  the  long  term,  and  result  in  degraded  performance.  On  the  other  hand,  if  operators 
realize  that  their  performance,  in  fact,  improves  with  the  alert,  then  they  may  accept  the  alerts 
more.  Future  research  should  address  how  operators  would  adapt  to  these  alerts  with  long  term 
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use.  In  this  study,  most  of  the  participants  liked  the  gradient  buzzing  of  the  continuous  feedback. 
Therefore,  one  possible  way  to  make  the  vibrating  wristband  more  acceptable  and  still  keep  its 
benefits  to  performance  may  be  to  design  the  wristband  to  not  continually  buzz,  but  have  a 
gradient  buzz  after  a  threshold  course  deviation  point  is  exceeded. 

Comments  from  the  participants  were  generally  positive  about  having  haptic  cues  complement 
the  visual  display.  Some  participants  even  said  that  they  used  the  haptic  cues  as  the  primary 
agent  to  identify  an  off-nominal  UAV  state,  and  did  the  visual  check  only  after  they  received  a 
haptic  feedback.  The  positive  attitude  towards  these  feedback  types  is  promising  for  operator 
acceptance,  but  also  is  concerning  especially  if  operators  start  depending  on  feedback 
inappropriately.  With  such  an  overreliance  on  feedback,  system  failures  can  lead  to  hazardous 
situations  that  may  not  occur  otherwise. 

Even  if  the  positive  subjective  responses  seem  to  indicate  that  the  operators  would  benefit  from 
haptic  feedback  in  general,  the  lack  of  a  baseline  condition  in  this  experiment  (i.e.,  no  feedback) 
makes  it  hard  to  make  such  a  claim.  This  experiment  focuses  on  comparing  continuous  haptic 
feedback  to  discrete  haptic  feedback.  In  order  to  assess  the  true  benefits  of  haptic  feedback, 
baseline  data  should  be  collected  where  participants  perform  the  same  tasks  without  the 
assistance  of  feedback.  Moreover,  a  direct  comparison  between  auditory  and  haptic  feedback  is 
also  needed.  A  future  study  focusing  on  these  issues  with  a  more  representative  participant 
population  can  provide  better  insights  into  how  peripheral  feedback  can  be  implemented  in  UAV 
supervisory  control. 
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Abstract 


A  future  implementation  of  unmanned  aerial  vehicle  (UAV)  operations  is  having  a  single 
operator  control  multiple  UAVs.  The  research  presented  here  explores  possible  avenues  of 
enhancing  audio  cues  of  UAV  interfaces  for  this  futuristic  control  of  multiple  UAVs  by  a  single 
operator.  This  project  specifically  evaluates  the  value  of  continuous  and  discrete  audio  cues  as 
indicators  of  course  deviations  or  late  arrivals  to  targets  for  UAV  missions.  It  also  looks  at  the 
value  of  the  audio  cues  in  single  and  multiple  UAV  scenarios. 

To  this  end,  an  experiment  was  carried  out  on  the  Multiple  Autonomous  Unmanned  Vehicle 
Experimental  (MAUVE)  test  bed  developed  in  the  Humans  and  Automation  Laboratory  at  the 
Massachusetts  Institute  of  Technology  with  44  military  participants.  Specifically,  two  continuous 
audio  alerts  were  mapped  to  two  human  supervisory  tasks  within  MAUVE.  One  of  the 
continuous  audio  alerts,  an  oscillating  course  deviation  alert  was  mapped  to  UAV  course 
deviations  which  occurred  over  a  continual  scale.  The  other  continuous  audio  alert  tested  was  a 
modulated  late  arrival  alert  which  alerted  the  operator  when  a  UAV  was  going  to  be  late  to  a 
target.  In  this  case  the  continuous  audio  was  mapped  to  a  discrete  event  in  that  the  UAV  was 
either  on  time  or  late  to  a  target.  The  audio  was  continuous  in  that  it  was  continually  on  and 
alerting  the  participant  to  the  current  state  of  the  UAV.  It  either  was  playing  a  tone  indicating 
the  UAV  was  on  time  to  a  target  or  playing  a  tone  indicating  the  UAV  was  late  to  a  target.  These 
continuous  alerts  were  tested  against  more  traditional  single  beep  alerts  which  acted  as  discrete 
alerts.  The  beeps  were  discrete  in  that  when  they  were  used  for  monitoring  course  deviations  a 
single  beep  was  played  when  the  UAV  got  to  specific  threshold  off  of  the  course  or  for  late 
arrivals  a  single  beep  was  played  when  the  UAV  became  late. 

The  results  show  that  the  use  of  the  continuous  audio  alerts  enhances  a  single  operator’s 
performance  in  monitoring  single  and  multiple  semi-autonomous  vehicles.  However,  the  results 
also  emphasize  the  necessity  to  properly  integrate  the  continuous  audio  with  the  other  auditory 
alarms  and  visual  representations  in  a  display,  as  it  is  possible  for  discrete  audio  alerts  to  be  lost 
in  aural  saliency  of  continuous  audio,  leaving  operators  reliant  on  the  visual  aspects  of  the 
display. 
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Introduction 


In  current  military  conflicts,  the  unmanned  aerial  vehicle  (UAV)  is  one  of  the  most  highly  sought 
after  air  assets  (Whitney,  2007).  UAVs  provide  real  time  information  and  support  to  battlefield 
operations  remotely  from  either  local  or  global  control  station  platforms.  Each  of  these 
operational  environments  presents  fluid  atmospheres  of  noise  and  activity. 

Currently  many  of  the  medium  and  high  altitude  UAV  operations  require  multiple  operators  to 
control  a  single  UAV;  the  desire  for  the  future  is  to  invert  the  numbers  and  have  a  single  operator 
controlling  multiple  UAVs  (Culbertson,  2006).  To  achieve  this,  research  is  needed  to  understand 
how  best  to  aid  and  integrate  the  human  operator  into  a  system  of  semi-autonomous  UAVs.  Due 
to  potential  information  overload  of  operators  in  visually  saturating  domains  such  as  UAV 
supervisory  control,  reducing  the  workload  by  offloading  some  cognitive  processing  to  the  other 
senses,  primarily  auditory  and  haptic,  is  critical.  The  point  of  this  research  is  to  explore  possible 
auditory  enhancements  that  can  reduce  operator  workload  and  increase  operator  performance. 


Background 

Improving  UAV  operator  performance  via  audio  enhancements  has  been  an  active  area  of 
research,  particularly  in  the  areas  of  spatial  audio,  which  has  been  shown  to  aid  UAV  operators 
in  target  recognition,  and  3D  audio,  which  has  been  shown  to  enhance  alert  and  warning 
detection  (for  a  review  of  the  literature,  see  (Nehme  &  Cummings,  2006)).  One  area  in  the  audio 
domain  that  has  received  less  attention  is  the  use  of  continuous  audio  peripheral  displays, 
particularly  in  complex,  time  pressured  domains  such  as  UAV  operation  in  command  and  control 
settings. 

Research  in  the  medical  domain  indicates  that  continuous  auditory  displays  (also  known  as 
sonifications)  can  aid  anesthesiologists  in  monitoring  patient  status  (Watson  &  Sanderson,  2004). 
Deviations  from  normal  breathing  patterns  matched  to  musical  notes  allowed  doctors  to  aurally 
detect  a  difference  in  patient  status,  while  visually  focusing  on  other  tasks.  Given  that  such  an 
approach  has  worked  in  this  medical  monitoring  domain,  it  is  not  clear  whether  continuous 
auditory  displays  can  be  applied  to  a  command  and  control  setting,  particularly  for  the 
supervisory  control  of  one  or  more  unmanned  vehicles  by  a  single  operator. 

While  continuous  audio  displays  can  allow  operators  to  effectively  divide  attention  so  that 
disparate  information  can  be  taken  in  through  different  channels  (such  as  visual  and  aural),  the 
environment  plays  a  major  role.  In  the  case  of  the  anesthesia  study,  operating  rooms  are 
generally  not  as  noisy  as  those  settings  typically  found  in  command  and  control  environments. 
For  example,  Shadow  UAV  operators  must  work  in  the  back  of  a  truck  with  significant  noise  due 
to  nearby  generators.  In  addition,  unlike  an  operating  room,  UAV  operators  wear  headsets  so  that 
they  can  communicate  with  both  inter  and  intra  team  members.  Also  unlike  the  operating  room, 
which  may  be  connected  to  an  intercom  in  another  room,  there  is  significant  communication 
coming  into  UAV  operators  from  the  outside  world.  UAV  operators  must  contend  with  an  air 
traffic  control-like  environment,  constantly  scanning  the  audio  landscape  for  possible  incoming 
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communications,  as  well  as  listening  to  general  radio  chatter  in  order  to  gain  situational 
awareness  for  the  overall  mission. 

Given  these  significant  UAY  environmental  constraints,  which  are  tantamount  to  noisy  working 
conditions,  the  design  of  an  audio  display,  particularly  a  continuous  one,  is  difficult.  Moreover,  a 
sonification  should  map  to  the  task,  i.e.  perceptual  changes  should  accurately  match  the  change 
in  the  data  (Barrass  &  Kramer,  1999).  Thus,  discrete  events  should  map  to  discrete  signals,  and 
continuous  events  should  map  to  a  continuous  auditory  display.  In  addition,  in  high  workload 
environments,  whether  sonifications  can  improve  performance,  or  whether  operators  will  ignore 
the  auditory  displays  and  rely  primarily  on  the  visual,  is  not  clear.  In  order  to  investigate  these 
issues,  an  experiment  was  conducted  which  is  detailed  in  the  remainder  of  the  paper. 


The  MAUVE  Simulator 

The  Massachusetts  Institute  of  Technology’s  Humans  and  Automation  Laboratory  (HAL) 
developed  the  Multiple  Autonomous  Unmanned  Vehicle  Experimental  (MAUVE)  test  bed, 
which  is  a  two  screen  interface  that  allows  the  simulation  of  any  number  of  unmanned  vehicles 
conducting  strike  operations  in  hostile  environments.  MAUVE,  for  this  experiment,  allowed  the 
operator  to  arm  and  fire  on  targets  with  various  priorities  while  monitoring  each  UAV’s  flight  of 
path,  event  timeline,  and  radio  traffic. 

The  test  bed  provided  a  map  for  geo-spatial  tracking  of  UAVs  on  their  preset  flight  paths  along 
with  an  interaction  control  panel  for  making  UAV  control  inputs  (Map  Display,  Figure  1). 
Control  inputs  included  arming  and  firing  a  UAV  to  destroy  targets,  directing  specific  path 
changes,  and  acknowledging  radio  calls.  The  path  changes  included  commanding  a  UAV  to  skip 
a  target  to  get  a  UAV  back  on  schedule.  The  operator  could  also  reset  the  UAV’s  onboard 
navigation  to  get  the  vehicle  back  on  course  if  it  drifted. 

A  timeline  for  each  UAV,  a  scheduling  decision  support  tool  for  each  UAV,  and  chat  interfaces 
were  also  provided  (Timeline  Display,  Figure  1).  This  display  helped  operators  ensure  UAVs 
would  be  on  time  to  targets  by  monitoring  when  they  would  arrive  at  targets,  and  in  what  phase 
they  were  currently  located.  In  particular,  the  timeline  used  the  color-coding  provided  in  Table  1 
to  represent  phase  of  flight. 
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Table  1:  UAV  Color-Coded  Flight  Phases 


UAV  Action 

Color 

Enroute 

Gray 

Loitering 

Blue 

Arming  Payload 

Yellow 

Firing  Payload 

Orange 

Return  to  Base 

Green 

The  general  task  for  MAUVE  operators  for  this  experiment  was  to  monitor  one  or  more  UAVs 
on  a  preplanned  mission  involving  several  waypoints  and  targets.  The  primary  task  of  the 
operator  was  to  ensure  weapons  were  dropped  on  the  correct  targets  during  the  specified 
windows  of  opportunity.  However,  unexpected  head  winds  or  crosswinds  could  cause  UAVs  to 
slow  their  speed  or  drift  off  course,  requiring  the  operator  to  take  corrective  action  for  projected 
target  late  arrivals  or  course  deviations.  Late  arrivals  were  a  higher  priority  than  course 
deviations.  In  addressing  late  arrivals,  the  priority  of  the  target  (high,  medium,  or  low)  also  had 
to  be  addressed,  such  that  it  was  more  important  to  hit  a  high  priority  target  than  a  medium 
priority  target,  and  so  on.  In  addition,  operators  also  had  to  ensure  the  UAVs  would  return  to 
base  by  a  specified  time. 
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Bonification  Design  and  Integration 


Four  auditory  alerts  were  created  by  Charles  River  Analytics,  Inc.  for  incorporation  into 
MAUVE,  which  represent  experimental  independent  variables.  These  four  auditory  alerts  were 
the  1)  threshold  course  deviation  alert,  2)  the  oscillating  course  deviation  alert,  3)  the  threshold 
late  arrival  alert,  and  4)  the  modulated  late  arrival  alert.  These  four  auditory  alerts  were  designed 
to  test  the  differences  between  discrete  and  continuous  alarms  for  the  two  conditions  of  late 
arrivals  and  course  deviations.  The  threshold  alerts  were  designed  to  be  discrete  alarms;  they 
emitted  a  single  discrete  audio  alert  when  some  precondition  was  met.  The  oscillating  and 
modulated  alerts  were  designed  to  be  continuous  alarms;  they  provided  continual  audio 
information  about  the  state  of  a  particular  variable  within  MAUVE.  Instead  of  indicating  a  clear 
distinct  threshold,  the  continuous  and  modulating  alerts  gradually  increased  as  the  severity  of 
either  the  late  arrival  or  course  deviation  condition  increased. 

The  threshold  course  deviation  alert  consisted  of  a  single  beep  with  a  fundamental  frequency  of 
1000  Hz.  The  beep  lasted  ~8  ms  and  played  whenever  a  UAV  moved  completely  away  from  its 
plotted  course  line. 

The  oscillating  course  deviation  alert  represented  both  the  existence  and  severity  of  the  UAV 
course  deviations.  It  consisted  of  comb  filters  that  were  applied  to  a  mix  of  pink  noise  and  the 
ambient  signal.  Using  pink  noise  means  that  filters  equalized  the  energy  of  sound  at  each  octave 
level,  resulting  in  a  constant  level  presentation  of  the  audio.  The  mix  ranged  from  0.2  pink  noise 
for  low  deviation  to  0.9  pink  noise  for  high  deviation.  The  comb  filters  had  a  base  delay  of  0.2 
ms,  with  a  50  percent  each  mix  of  the  base  signal  and  a  feed  forward  delay.  The  delay  values 
were  then  oscillated  to  create  a  periodic  signal.  Because  this  was  a  continuous  audio  scheme,  it 
played  continually  to  provide  an  auditory  image  of  UAV  path  position.  As  a  UAV  drifted  further 
off  course,  the  frequency  of  oscillation  of  the  comb  filter  delay  decreased  from  17  Hz  to  4.5  Hz, 
and  the  depth  of  oscillation  increased  from  0.2  ms  to  0.7  ms,  thus  changing  on  a  continual  scale. 

The  threshold  late  arrival  alert  consisted  of  a  single  beep  with  a  fundamental  frequency  of  415 
Hz.  The  beep  lasted  ~18  ms,  and  played  whenever  a  UAV  was  projected  to  be  late  to  any  target. 
A  UAV  was  determined  late  to  a  target  when  it  had  exceeded  its  window  of  opportunity  to 
destroy  the  target  and  was  not  going  to  be  able  to  get  over  the  target  in  time  to  fire  on  the  target 
during  the  set  time  window  it  was  required  to  be  at  the  target. 

The  modulated  late  arrival  alert  consisted  of  discrete  harmonic  signals  continuously  playing  to 
indicate  when  a  projected  late  arrival  at  a  target.  These  sounds  played  continually  until  the 
operator  addressed  the  issue  with  the  corrective  action  of  choosing  to  delay  the  UAV,  skip  a 
target,  or  the  issue  disappeared  because  the  UAV  automatically  continued  past  the  target  when  it 
was  going  to  be  late  to  the  next  target.  The  harmonic  signals  were  composed  of  five  Formant 
filters  that  were  applied  to  a  mix  of  pink  noise  and  the  ambient  signal.  During  the  condition  of  no 
late  arrivals,  a  baseline  audio  condition  was  generated  with  two  filters  set  to  261.6  Hz,  two  filters 
set  at  329.4  Hz,  and  one  filter  set  at  392.0  Hz  (a  major  C  triad  with  extra  balance  in  the  root  and 
third).  If  it  was  late  to  a  low  priority  target,  a  signal  was  generated  with  two  filters  set  to  261.6 
Hz,  two  filters  set  at  31 1.1  Hz,  and  one  filter  set  at  392.0  Hz  (a  minor  C  triad  with  extra  balance 
in  the  root  and  third).  If  it  was  late  to  a  medium  priority  target,  a  signal  was  generated  with  three 
filters  set  to  293.6  Hz  and  two  filters  set  at  415.3  Hz  (a  tritone  interval  with  a  D  root).  If  the 
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UAV  was  predicted  to  be  late  to  a  high  priority  target,  a  signal  was  generated  with  three  filters 
set  to  369.9  Hz  and  two  filters  set  at  523.25  Hz  (a  tritone  interval  with  an  F#  root).  As  the 
priority  increased,  the  pink  noise  mix  also  increased,  from  0.25  for  the  baseline,  to  0.7,  0.8,  and 
1.0  for  the  three  priority  levels. 

All  audio  alerts  were  presented  equally  in  both  ears  through  the  Sensimetrics  HDiSP.  Pictured  in 
Figure  3,  the  HDiSP  is  an  over-the-head  headset  with  sealed  around-the-ear  ear  cups.  It  provides 
ambient  noise  attenuation  ranging  from  35  to  40  db  for  the  spectrum  between  500  and  8  kHz. 
The  headset  has  integrated  digital  signal  processors,  which  produce  the  signals  used  in  this 
experiment  based  on  generative  audio  filters  and  processing  of  the  ambient  signal,  which  was 
received  from  microphones  mounted  on  the  headband. 


Figure  13:  Sensimetrics  HDiSP. 

Experimental  Protocol 


Experiment  Objective 

The  objective  of  this  experiment  was  to  determine  whether  sonifications  maximize  the 
information  conveyed  to  UAV  operators  more  efficiently  than  typical  discrete  alarms  typically 
used  in  current  ground  control  stations.  In  addition,  the  impact  of  continuous  versus  discrete 
alerting  on  operators  attempting  to  control  multiple  UAVs  was  also  a  research  question. 


Research  Hypothesis 

There  were  three  null  hypotheses  explored  in  this  research: 

1.  For  various  combinations  of  discrete  and  continuous  alerts,  there  was  no 
difference  in  the  operator’s  performance. 

2.  For  operators  controlling  either  a  single  or  multiple  UAVs,  there  was  no 
difference  in  the  operator’s  performance. 

3.  There  was  no  interaction  between  the  discrete  and  continuous  alerts  combinations 
and  whether  the  operator  was  controlling  one  or  more  UAVs. 
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The  alternative  hypotheses  for  the  first  two  null  hypotheses,  therefore,  were  that  there  is  a 
difference,  and  for  the  third  null  hypothesis,  that  there  was  interaction. 


Task 

For  this  experiment,  the  operator  was  given  the  following  as  the  primary  objective  for  each 
mission: 

Make  sure  the  UAV(s)  maximize  the  number  of  targets  engaged  as  well  as  arrive  back  at 
the  base  safely. 

Further,  supervision  of  each  of  the  UAV(s)  for  each  mission  was  broken  down  into  the  following 
prioritized  sub-tasks,  from  highest  priority  to  lowest: 

7.  Return  to  base  (RTB)  within  the  time  limit  for  the  mission  (this  limit  was  clearly 
marked). 

8.  Comply  with  recovery  rules  for  course  deviations. 

9.  Comply  with  recovery  rules  for  target  late  arrivals. 

10.  Destroy  all  targets  before  their  time  on  target  (TOT)  window  ended. 

11.  Avoid  taking  damage  from  enemies  by  avoiding  all  threat  areas. 

12.  Acknowledge  all  “Push”  radio  calls. 

Participants  were  trained  to  follow  these  priorities,  and  the  objectives  and  priorities  above  were 
displayed  to  them  throughout  the  experiment.  Within  the  priorities,  recovery  rules  of  how  to 
recover  the  UAV  from  a  course  deviation  or  late  arrival  were  also  displayed.  See  Appendix  A 
for  a  description  of  a  course  deviation  or  late  arrival.  The  instructions  on  how  to  comply  with 
recovery  rules  for  course  deviations  were  to  click  “Reset  Navigation.”  This  theoretically  reset  the 
onboard  navigation  in  the  UAV  and  caused  it  to  fly  back  to  the  planned  flight  path. 

The  instructions  on  how  to  comply  with  recovery  rules  for  target  late  arrivals  were: 

•  Low  Priority  -  Click  “Skip  Target” 

•  Medium  Priority  -  Click  “Skip  Target”  or  Employ  the  decision  support  visualization 
(DSV). 

•  High  Priority  -  Employ  DSV  before  Requesting  Delay  or  Clicking  “Skip  Target” 

For  low  priority  targets,  clicking  the  “Skip  Target”  button  would  cause  the  UAV  to  proceed 
immediately  to  the  next  target.  For  medium  and  high  priority  targets,  the  option  was  given  to  use 
the  DSV,  which  was  a  tool  that  illustrated  the  effect  that  delaying  the  time  on  target  for  the  one 
late  target  had  on  the  remaining  targets  for  that  UAV’s  flight  plan.  With  the  assistance  of  the 
information  presented  in  the  DSV,  participants  then  decided  whether  they  should  click  the  “Skip 
Target”  button  or  “Request  Delay”  button.  If  they  clicked  “Request  Delay,”  they  then  had  to 
wait  to  see  if  a  delay  was  granted,  and  if  the  delay  was  granted  the  UAV  was  then  no  longer  late 
to  the  target  and  would  be  able  to  destroy  it.  For  a  high  priority  target  the  immediate  response 
was  to  employ  the  DSV. 
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Independent  Variables 


The  experiment  was  a  4x2  fixed  factor  repeated  measures  model,  with  two  independent 
variables:  the  audio  condition  (a  between-subjects  treatment),  and  the  number  of  vehicles  under 
control  (a  repeated  within- subjects  factor).  The  four  levels  of  the  audio  condition  factors  were 
combinations  of  the  four  auditory  alerts  described  earlier.  The  audio  conditions  were:  1)  the 
threshold  audio  condition  for  both  the  late  arrivals  and  course  deviations,  2)  the  continuous 
oscillating  course  deviation  audio  condition  with  threshold  alerts  for  the  late  arrivals,  3)  the 
continuous  modulated  late  arrival  audio  condition  with  a  threshold  alert  for  course  deviations, 
and  4)  the  combination  audio  condition  which  consisted  of  the  oscillating  course  deviation  alert 
and  the  modulated  late  arrival  alert. 

The  second  independent  variable,  the  number  of  vehicles  under  control,  had  two  levels:  single 
UAV  and  multiple  UAV.  In  the  single  level,  the  participant  supervised  only  one  UAV,  while  in 
the  multiple  factor  level,  the  participant  supervised  four  UAYs.  The  framework  of  this 
experiment  is  represented  in  Table  2. 

Table  2:  Experimental  Design. 


Scenario  (Repeated) 


Audio 

Scheme 

(Between) 


Single  UAV 

Multi  UAV 

Both  Threshold 

"" 

-- 

Continuous  Course  Deviation,  Threshold  Late  Arrival 

"" 

"" 

Continuous  Late  Arrival,  Threshold  Course  Deviation 

"" 

"" 

Combination 

Dependent  Variables 

Dependent  variables  included  the  reaction  time  to  correct  course  deviations,  the  number  of 
missed  course  deviations,  the  reaction  time  to  correct  projected  late  arrivals,  the  number  of 
missed  late  arrivals,  NASA  TLX  scores,  and  number  of  missed  radio  calls.  These  are  discussed 
in  detail  below. 

1 .  Course  deviation  reaction  time 

In  each  test  scenario,  participants  were  expected  to  respond  to  four  triggered  course  deviations. 
A  course  deviation  was  defined  as  when  a  UAV  no  longer  appeared  to  be  following  its  planned 
flight  path.  A  general  rule  for  defining  a  course  deviation  was  a  UAV  that  no  longer  appeared 
over  the  flight  path’s  line  or  a  UAV  completely  left  the  flight  path. 
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2.  Course  deviation  errors  of  omission 


Anytime  the  participant  failed  to  respond  to  one  of  four  triggered  course  deviations,  an  error  of 
omission  was  counted. 

3.  Late  arrival  reaction  time 

Four  late  arrivals  occurred  in  each  test  scenario.  Late  arrivals  were  caused  when  a  UAV  slowed 
down  because  of  unforeseen  headwinds  and  was  no  longer  able  to  reach  a  target  in  time  to 
complete  the  firing  mission. 

4.  Late  arrival  errors  of  omission 

Errors  of  omission  for  late  arrivals  occurred  when  the  participant  failed  to  respond  to  one  of  the 
four  triggered  late  arrivals. 

5.  Radio  calls  missed  (secondary  workload  assessment) 

As  a  secondary  workload  task,  the  number  of  missed  radio  was  an  indication  of  the  operator’s 
level  of  mental  workload.  The  count  of  missed  radio  calls  measured  spare  mental  capacity. 
Participants  were  instructed  to  monitor  a  recording  of  continual  air  traffic  radio  chatter  for  the 
word  “Push.”  The  word  “Push”  occurred  62  times  in  a  30  minute  session,  with  an  average  of  27 
seconds  between  each  “Push”  radio  call.  To  acknowledge  the  radio  call,  participants  clicked  an 
“Acknowledge  Push”  button  on  the  display. 

6.  NASA  TLX  score  (subjective  workload  assessment) 

The  NASA  Task  Load  Index  gathered  participants’  subjective  assessment  on  a  scale  of  1  to  20  of 
mental  demand,  physical  demand,  temporal  demand,  effort,  performance,  and  frustration.  A 
participant  was  then  asked  to  rate  these  six  dimensions  against  each  other  to  determine  their 
importance  in  the  participant’s  workload  (Hart  &  Staveland,  1988).  Since  there  was  no  physical 
demand  in  this  experiment,  each  participant  was  told  to  rate  the  physical  demand  so  that  it  was 
zeroed  out  of  the  score.  No  difference  between  the  single  and  multiple  UAV  scenario  was 
expected  because  the  scenarios  were  designed  with  equal  workloads.  Appendix  B  shows  how 
the  single  and  multiple  UAV  scenarios  were  designed  with  the  same  events.  The  only  difference 
was  that  the  multiple  UAV  scenario  has  the  events  divided  over  4  UAVs  while  the  single 
scenario  has  them  all  occurring  with  1  UAV. 


Apparatus 

The  experiment  was  administered  in  a  testing  room  with  a  background  ambient  noise  level  of 
~64dB  on  a  C-scale.  MAUVE  was  run  on  a  multi-modal  workstation  (MMWS).  Pictured  in 
Figure  2,  the  MMWS  is  a  four  screen  computer  display. 
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Figure  14:  The  Multi-Modal  Workstation  (MMWS). 


The  three  screens  across  the  top  were  21  inches  and  set  to  display  at  1280  x  1024  pixel,  16  bit 
color  resolution,  and  the  15  inch  bottom  center  screen  was  set  at  1024  x  768  pixels,  32  bit  color 
resolution.  The  computer  used  to  run  the  simulator  was  a  Dell  Optiplex  GX280  with  a  Pentium  4 
processor  and  an  Appian  Jeronimo  Pro  4-Port  graphics  card.  Participants  controlled  the  simulator 
through  a  generic  corded  computer  mouse.  Throughout  the  experiment,  the  top  left  display 
showed  the  mission  objectives  and  priorities,  and  the  bottom  center  displayed  the  color-coding 
for  MAUVE  (Table  1).  The  top  center  display  contained  the  left  MAUVE  map  and  interaction 
display,  and  the  top  right  display  included  the  right  MAUVE  timeline  and  decision  support 
display  (Figure  1).  During  the  test  sessions,  the  screen  recordings  for  the  left  and  right  MAUVE 
displays  were  captured  by  Camtasia  ®  recording  software. 


Participants 

Forty-four  military  personnel  were  paid  $10  an  hour  to  participate.  The  experiment  took 
anywhere  from  2.5  to  3  hours  to  complete.  Participants’  ages  ranged  from  20  years  to  42  years, 
with  an  average  age  of  26  years.  There  were  3  Navy  midshipmen,  1  Army  reservist  specialist,  1 
Air  Force  staff  sergeant,  and  39  officers  from  the  Army/Navy/ Air  Force.  Overall,  the  personnel 
tested  had  a  combined  experience  of  over  250  years  of  active  duty  military  service,  with  each 
member  having  an  average  of  5.8  years  of  active  duty  service.  Five  pilots  contributed  as  test 
participants,  and  most  of  the  28  junior  officers  tested  will  be  future  military  pilots. 

For  the  data  analysis,  5  participants  were  dropped  because  of  problematic  data.  The  first  two 
were  omitted  because  of  the  failure  of  the  test  proctor  to  administer  proper  training.  The  third 
was  dropped  because  of  a  3  day  interruption  between  the  single  and  multiple  UAV  test  sessions. 
The  fourth  participant  was  not  used  because  his  secondary  data  was  an  outlier  for  both  the  single 
and  multiple  UAV  scenarios.  The  fifth  participant  dropped  was  an  active  duty  Air  Force 
lieutenant  colonel  with  4  years  experience  as  a  maintenance  officer  and  an  additional  4  years  of 
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flight  line  time  as  a  pilot  and  flight  test  engineer.  He  reported  having  been  diagnosed  with 
tinnitus  and  loss  of  high  frequency  tone  in  one  ear.  During  the  audio  training,  he  reported  not 
being  able  to  hear  the  differentiation  between  a  low  and  high  course  deviation  with  the 
oscillating  course  deviation  alert.  His  hearing  test  results  indicated  age  induced  hearing  loss. 


Procedure 

Each  participant  experienced  three  separate  consecutive  phases  in  completing  the  experiment. 
Each  had  a  60  to  70  minute  training  session  followed  by  a  70  to  80  minute  test  session,  and  then 
a  10  minute  post-test  survey.  For  the  training  session,  each  participant  completed  a  participant 
consent  form  and  demographics  survey,  received  standardized  training  from  a  PowerPoint® 
tutorial,  a  hearing  test,  audio  training  for  the  audio  condition  they  were  presented,  and  a  thirty 
minute  practice  test  session  with  a  trial  run  through  of  NASA  TLX.  If  required,  a  participant 
could  do  an  additional  10  minute  practice  session  to  ensure  understanding  of  the  proper  MAUVE 
control  actions.  Copies  of  the  participant  consent  form,  demographics  survey,  tutorial,  and  post¬ 
test  survey  are  in  Appendices  C,D,  E,  and  F  respectively. 

The  hearing  test  was  designed  to  identify  whether  the  participant  suffered  any  hearing  loss 
(either  temporary  or  permanent).  For  the  hearing  test,  participants  listened  to  a  CD  of  test  tones 
from  40dBSPL  down  to  16dBSPL,  with  5  frequencies  tested  (500  Hz,  1  kHz,  2  kHz,  4  kHz,  and 
8  kHz).  The  listener  acknowledged  how  many  beeps  they  heard  for  each  ear  and  each  frequency. 
The  range  tested  ensured  they  could  hear  to  within  3dB  of  the  noise  floor  of  the  headset  in  the 
normal  lab  conditions.  Based  on  the  method  of  descending  limits,  the  test  was  designed  so  that 
hearing  5  beeps  or  more  at  each  frequency  showed  adequate  hearing  (Snodgrass,  1975;  Wundt, 
1902)  If  fewer  beeps  were  heard  at  8  kHz  than  at  1  kHz,  participant  exhibited  high  frequency 
loss  (Cooper  &  Owen,  1976;  Humes,  Joellenbeck,  &  Durch,  2006).  Age  related  permanent  loss 
showed  up  most  at  high  frequencies  (i.e.  8  kHz  congestion  and  other  losses  show  up  at  lower 
frequencies  and  across  range)  (Walden,  Prosek,  &  Worthington,  1975).  The  hearing  test  was 
generated  by  Dr.  Thomas  Edward  von  Wiegand  of  Sensimetrics  Corporations,  in  conjunction 
with  the  HDiSP. 

The  audio  training  gave  each  participant  exposure  to  all  of  the  audio  alerts  they  would  hear. 
They  were  presented  with  their  respective  late  arrival  and  course  deviation  alerts,  as  well  as  the 
radio  chatter.  After  introducing  each  of  these  individually,  they  were  played  together  to  give  the 
full  effect.  After  a  demonstration  of  the  audio  condition,  participants  were  encouraged  to  play 
with  the  audio  demonstration  software  until  they  were  comfortable  with  recognizing  each  of  the 
sounds. 

After  the  training  was  completed,  each  participant  then  completed  both  the  single  UAV  scenario 
and  the  multiple  UAV  scenario.  The  experiment  was  counterbalanced;  half  of  the  participants 
completed  the  single  UAV  scenario  first,  and  the  other  half  finished  the  multiple  UAV  scenario 
first.  Additionally,  the  participants  were  randomly  assigned  to  the  four  audio  schemes  to 
maintain  a  balanced  study.  At  the  conclusion  of  every  test  session,  MAUVE  generated  a  data 
log,  with  all  the  previously  discussed  dependent  variables.  Additionally,  each  participant 
completed  a  NASA  TLX  following  each  test  session.  At  the  conclusion  of  the  test  session,  each 
participant  completed  a  post-test  survey,  was  paid,  and  thanked  for  their  time  and  involvement. 
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Results 


For  statistical  analysis,  repeated  measures  ANOVAs  were  conducted  for  each  dependent 
variable.  A  summary  of  the  dependent  variables  with  significant  results  is  presented  in  Table  3. 
The  audio  scheme  factor  is  a  between-subjects  factor  for  the  four  audio  conditions  and  the 
scenario  is  a  repeated  within-subjects  factor  of  single  versus  multi  UAV  levels.  The  current  data 
analysis  is  based  on  39  participants,  which  meets  the  23  participant  requirement  of  the  a  priori 
power  analysis  for  a  power  of  0.80.  Excluding  the  late  arrival  reaction  times  and  error  of 
omission  counts,  all  of  the  remaining  variables  analyzed  met  normality  and  homogeneity 
assumptions.  While  a  =  0.05,  p  values  between  0.05  and  0.10  are  considered  marginally 
significant.  See  Appendix  G  for  further  details  on  the  statistical  analysis  tests. 

Table  3:  Summary  of  experiment  statistical  results. 


Dependent  Variable 

Audio 

Scheme 

Scenario 

Interaction 

Course  Deviation 
Reaction  Times 

F(3,35)=2.878 

p=.05 

F(1 ,35)=3.215 

p=.08 

NS 

Late  Arrival  Reaction 
Times 

F(3,35)=3.345 

p=.03 

F(1 ,35)=20.737 

p<.001 

NS 

1 .  Course  deviation  reaction  time 

Figure  9  shows  the  means  for  the  course  deviation  reaction  times  across  the  four  audio 
conditions  (combination,  continuous  late  arrival,  continuous  course  deviation,  and  threshold). 
The  omnibus  test  shows  there  is  a  statistically  significant  difference  in  the  performance  of 
participants  based  on  the  audio  scheme  (F(3,35)=2.878,  p=.05),  and  a  marginally  significant 
difference  due  to  the  scenario  (F(l,35)=3.215,  p=.08).  A  Tukey  post  hoc  comparison  shows  that 
the  primary  points  of  difference  are  for  the  combination  to  threshold  audio  conditions  for  the 
single  UAV  scenario  (p=0.02).  There  is  not  a  significant  difference  between  any  of  the  other 
points. 
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Figure  15:  The  treatment  means  plots  for  the  course  deviations  reaction  times. 


In  Figure  9,  there  is  an  apparent  division  in  the  data  based  on  which  audio  alert  is  present  for  the 
course  deviation.  As  part  of  the  post  hoc  analysis  based  on  the  data  clustering  in  Figure  4,  the 
model  was  reconfigured  as  a  2x2  fixed  factor  repeated  measures  model  with  audio  scheme  and 
scenario  as  the  independent  variables,  respectively  (Table  4).  Instead  of  the  four  audio 
conditions,  the  factor  levels  were  collapsed  into  two  levels:  those  exposed  to  the  oscillating 
course  deviation  alert  and  those  exposed  to  the  threshold  course  deviation  alert.  The  audio 
scheme  remained  still  a  between-subjects  factor  while  the  scenario  remained  the  same  as  a 
repeated  within-subjects  factor. 

Table  4:  Post  Hoc  Audio  Alert  Experimental  Design 


Audio  Scheme 
(Between) 


Scenario  (Repeated) 


Single  UAV 

Multiple  UAV 

Oscillating  Course 
Deviation  Alert 

Threshold  Course 
Deviation  Alert 

_ 

_ 

The  combination  and  continuous  course  deviation  audio  conditions  use  the  oscillating  course 
deviation  alert  condition  while  the  threshold  course  deviation  alert  condition  is  used  in  the 
continuous  late  arrival  and  threshold  audio  conditions.  As  seen  in  Figure  16,  the  analysis 
confirms  that  the  participants  with  the  continuous  oscillating  alert  condition  performed 
significantly  better  than  those  with  the  threshold  course  deviation  alert  condition  (F(l,37)  = 
8.874,  p=.01). 
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Figure  16:  The  treatment  means  plots  for  the  two  alert  conditions  for  the  course  deviations  reaction  times. 

2.  Course  deviation  errors  of  omission 

Out  of  the  156  course  deviations  presented  to  the  39  participants,  9  were  missed  (5.8%).  The 
data  is  not  normal  and  the  non-parametric  tests  (Wilcoxon-Signed  Rank,  Kruskal- Wallis)  show 
no  significant  differences. 

3.  Late  arrival  reaction  time 

The  original  late  arrival  reaction  times  are  not  normal,  thus  a  natural  log  transformation  was 
performed.  The  omnibus  results  show  a  significant  difference  in  the  performance  of  participants 
between  the  four  audio  conditions  (F(3,35)=3.345,  p=.03)  and  a  significant  difference  due  to  the 
scenario  (F(l,35)=20.737,  p<.001). 

As  seen  in  Figure  10,  there  is  an  apparent  split  in  the  data.  A  post  hoc  Tukey  comparison  reveals 
there  is  significant  and  marginally  significant  difference  between  the  data  for  the  continuous 
course  deviation  audio  condition  and  the  other  three  audio  conditions:  threshold  audio  condition 
(p=.05),  continuous  late  arrival  audio  condition  (p=.08),  and  combination  audio  condition 
(p=.07).  Point-to-point  comparisons  reveal  the  difference  in  the  audio  scheme  is  not  significant 
for  any  of  the  single  UAV  comparisons,  but  for  the  multiple  UAV  audio  comparisons,  there  is  a 
significant  difference  between  the  combination  and  continuous  course  deviation  audio  conditions 
(p=.02),  the  continuous  course  deviation  and  continuous  late  arrival  audio  conditions  (p=.06), 
and  the  continuous  course  deviation  and  threshold  audio  conditions  (p=.07).  In  addition,  there  is 
a  significant  difference  in  performance  from  the  single  to  the  multiple  UAV  scenarios  for  the 
combination  audio  condition  (p=.04)  and  the  continuous  course  deviation  audio  condition 
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(p=.03).  These  post  hoc  comparisons  confirm  the  apparent  differences  within  the  treatment 
means  plot  in  Figure  10. 


Audio_Scheme 

- Combination 

Continuous 
Late  Arrival 
Continuous 
Course 
Deviation 
—  —  Threshold 


Figure  17:  The  treatment  means  plots  for  the  transformed  (LN)  late  arrival  reaction  times. 

In  fact,  when  the  combination,  continuous  late  arrival,  and  threshold  audio  conditions  are 
combined  as  one  condition  and  compared  to  the  continuous  course  deviation  audio  condition  the 
results  show  a  significant  difference  based  on  audio  condition  (F(l,37)  =  10.528,  p<.001)  and 
scenario  (F(l,37)  =  23.730,  p<.001). 

4.  Late  arrival  errors  of  omission 

All  39  participants  responded  to  all  of  the  4  late  arrivals  presented  to  each  of  them,  thus  there 
were  no  late  arrival  errors  of  omission. 

5.  Radio  calls  missed  (secondary  workload  assessment) 

For  the  number  of  missed  radio  calls,  there  is  no  significant  difference  between  the  audio 
conditions  or  across  the  scenarios.  According  to  this  secondary  workload  measurement,  the 
participants  appear  to  be  equally  work  saturated  regardless  of  their  assigned  audio  condition,  or 
whether  they  supervised  a  single  or  multiple  UAVs.  The  box  plots  in  Figure  11  show  larger 
variations  of  performance  in  the  multi  UAV  scenario  as  compared  to  the  single  scenario  for  all 
audio  conditions  except  the  threshold  audio  condition. 
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□  Single  UAV 

□  Multi  UAV 


Combination  Continuous  Late  Continuous  Threshold 
Arrival  Course 
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Audio  Scheme 


Figure  18:  The  box  plots  of  the  number  of  missed  radio  calls 

6.  NASA  TLX  score  (subjective  workload  assessment) 

Similar  to  the  secondary  workload  measurement,  the  subjective  measurement  of  the  NASA  TLX 
scores  are  not  significantly  different  for  the  audio  condition  or  scenario.  Because  these  are 
subjective  scores,  this  tool  is  best  for  comparing  within-subjects  across  the  scenarios,  so  this 
result  is  not  surprising. 


Discussion 

The  results  of  this  experiment  show  that  sonifications  for  two  different  events,  the  combination 
audio  scheme,  promoted  the  best  performance  for  the  two  reaction  time  dependent  variables. 
This  combination  scheme  consisted  of  oscillating  course  deviations  alerts  and  modulated  late 
arrival  alerts,  and  as  will  be  discussed  below,  their  integration  was  just  as  critical  as  their 
existence  for  the  individual  alerts. 

The  post  hoc  analysis  shows  that  the  oscillating  course  deviation  alert  is  beneficial  in  controlling 
both  a  single  and  multiple  UAVs.  Overall,  participants  were  quicker  to  respond  to  course 
deviations  with  the  oscillating  course  deviation  alert  than  with  the  threshold  course  deviation 
alert.  This  was  true  when  a  continuous  alert  was  used  either  just  for  the  course  deviation 
scenario  as  well  as  when  combined  with  the  late  arrival  continuous  alert.  One  reason  for  the 
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longer  course  deviation  reaction  times  for  the  threshold  alert  may  be  that  participants  who  have 
visually  identified  the  course  deviation  may  wait  for  the  threshold  alert  as  confirmation.  With  the 
oscillating  course  deviation,  the  auditory  confirmation  comes  sooner  as  they  integrate  the 
continuous  audio  with  the  visual  display.  One  participant  reported  doing  this.  On  the  whole, 
most  participants  reported  relying  on  the  audio  as  their  primary  alert.  They  did  not  even  visually 
monitor  for  a  course  deviation  until  they  were  alerted  by  the  audio.  The  better  performers, 
therefore,  were  the  participants  with  the  oscillating  course  deviation  alert  because  they  were 
forced  to  integrate  the  audio  feedback  continually  and,  as  a  result,  were  more  actively  visually 
monitoring  UAV  progression  on  or  off  course. 

The  results  for  the  late  arrival  reaction  do  not  show  the  same  trends.  Whether  participants  had  a 
continuous  modulated  alert  for  late  arrival  events  across  both  the  single  and  multiple  UAV 
conditions  did  not  make  a  difference  in  late  arrival  reaction  times.  However,  those  participants 
with  the  continuous  course  deviation  audio  condition  and  the  threshold  alert  for  late  arrivals 
performed  significantly  worse  in  terms  of  late  arrival  reaction  time  across  both  single  or  multiple 
UAVs  scenarios.  This  is  notable  because  when  presented  with  threshold  alerts  for  both  the 
course  deviation  and  late  arrival  events,  participants  responded  more  quickly  to  late  arrival  alerts 
than  when  the  continuous  course  deviation  alert  was  used. 

This  conflicting  result  could  be  indicative  of  an  aural  saliency  problem.  It  appears  the  oscillating 
course  deviation  audio  alert  may  saturate  the  audio  channel  to  a  point  that  the  late  arrival’s 
threshold  beep  alerts  are  not  heard,  resulting  in  longer  late  arrival  reaction  times.  Interestingly, 
the  combination  audio  condition  with  continuous  alerts  for  both  the  late  arrival  and  course 
deviation  conditions  produced  statistically  better  reaction  times  than  when  just  the  course 
deviation  continuous  alert  was  used.  The  integration  scheme  of  the  two  continuous  alerts  could 
explain  this  result.  The  continuous  modulated  late  arrival  alert  continues  to  sound  until  the 
participant  addresses  it;  the  participant  cannot  easily  miss  this  alert,  even  in  the  presence  of  the 
oscillating  course  deviation  alert.  In  contrast,  the  late  arrival  threshold  alert  sounds  once  and  is 
easily  missed  in  the  presence  of  the  continuous  course  deviation  alert.  This  was  noted  by  many 
participants  in  the  post  experiment  survey,  where  16  participants  asked  for  alerts  with  “more  than 
just  a  simple  beep”  but  instead  wished  for  a  more  repetitive  alert  that  will  “last”  so  they  will  have 
to  address  it. 

In  terms  of  error  rates  in  responding  to  both  course  deviation  and  late  arrival  events,  only  a  very 
small  percentage  of  course  deviation  alerts  were  missed,  and  all  of  the  late  arrivals  were 
responded  to  by  the  participants.  Both  events  were  also  represented  visually  which  demonstrates 
the  value  of  coding  on  multiple  channels.  One  difference  in  the  visual  alert  representation  that 
could  cause  this  slight  difference  in  error  rates  is  that  the  late  arrival  visual  alert  is  easily  seen  by 
the  physical  displacement  and  color  change  of  the  target  icon.  In  contrast,  whether  a  UAV  is 
truly  off  course  and  not  correcting  is  a  judgment  call  until  the  UAV  is  clearly  off  the  path. 
Appendix  A  contains  images  of  the  late  arrival  and  course  deviation  representations.  In  the 
simulation,  if  a  participant  did  not  recognize  and  correct  an  off  course  error  in  30  seconds,  the 
UAV  automatically  resumed  its  correct  course. 

In  addition  to  the  audio  condition  independent  variable,  the  impact  of  workload  was  examined. 
Objectively,  the  secondary  task  results  show  that  workload  appears  to  be  the  same  between  the 
four  audio  conditions  and  the  two  scenarios.  This  workload  equality  is  not  a  surprise,  because  as 


17 


discussed  earlier  and  as  shown  in  Appendix  B,  the  two  scenarios  were  designed  to  have 
comparable  workloads,  with  the  multi  UAV  scenario  dividing  the  monitoring  tasks  over  four 
vehicles  instead  of  just  one.  The  subjective  ratings,  the  NASA  TLX  data,  confirm  that  the 
participants  felt  the  two  scenarios  were  comparable  in  workload. 

However,  while  participants  observed  that  audio  alerts  were  “very  helpful”  when  focusing  on 
other  tasks,  it  was  “harder  to  comprehend  with  four  UAVs  than  one.”  The  results  generally 
support  these  comments.  Across  both  the  course  deviation  and  late  arrival  reaction  times, 
participants  reacted  more  quickly  to  the  single  UAV  scenario  than  the  multiple  UAV  case.  This 
result  is  not  surprising  since  this  trend  is  seen  throughout  air  traffic  controller  studies,  and  is  a 
known  effect  of  increased  cognitive  complexity  (Cummings  &  Tsonis,  2005).  As  the  number  of 
aircraft  that  require  supervisory  control  increases  (akin  to  the  number  of  UAVs  under  control), 
reaction  to  the  various  tasks  degrades  because  the  user’s  attention  is  divided  across  multiple 
entities. 

Comments  from  the  participants  were  generally  positive  about  having  auditory  cues  complement 
the  visual  display.  They  reported  that  the  audio  “helped  draw  my  attention  to  the  matter  so  I 
could  make  a  decision.”  However,  not  all  subjective  comments  were  positive.  Many 
participants,  including  actual  Air  Force  pilots,  reported  that  they  do  not  want  to  listen  to 
continuous  “noise  for  hours  at  a  time  day  after  day.”  A  few  participants  commented  that  the 
perceived  noise  made  them  feel  “overwhelmed”  and  was  “fatiguing.”  A  few  suggested 
“dampening  or  eliminating”  the  “constant  background  noise  when”  the  UAVs  operate  within 
normal  parameters.  It  is  important  to  note  that  these  scenarios  were  30  minute  test  scenarios,  and 
actual  UAV  operations  occur  over  much  longer  periods.  Integrating  this  audio  alert  into 
operations  must  be  balanced  against  what  operators  will  subjectively  tolerate.  It  is  possible  that 
continuous  audio  should  only  be  used  over  limited  time  spans,  during  high  risk  and  high 
workload  periods  of  the  mission.  These  are  areas  of  possible  future  research. 


Conclusion 

This  research  demonstrates  that  sonifications  can  help  single  operators  supervise  multiple  semi- 
autonomous  vehicles.  However,  auditory  cues  must  be  integrated  in  a  fashion  that  the  operators 
will  be  able  to  tolerate  subjectively,  and  these  signals  must  also  be  integrated  with  other  audio 
cues  to  ensure  balanced  aural  saliency.  This  research  also  showed  that  while  workload  is  an 
important  factor  in  overall  performance,  the  sonifications  were  equally  effective  for  participants 
controlling  one  or  four  UAVs. 
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APPENDIX  A  COURSE  DEVIATION  AND  LATE  ARRIVAL  IMAGES 


COURSE  DEVIATION 

The  green  line  in  the  image  below  indicates  the  plotted  and  desired  flight  path.  Anytime  the  UAV  has 
completely  departed  from  the  green  course  line  it  is  considered  to  have  deviated.  During  the  simulation, 
the  UAV  constantly  fluctuates  back  and  forth  on  the  flight  path,  but  participants  are  trained  to  respond 
when  the  UAV  actually  is  no  longer  touching  the  course  line.  Below  are  images  illustrating  UAV  3  on 
course,  almost  off  course,  and  off  course. 


On  Course  Almost  Off  Course  Off  Course 


LATE  ARRIVAL 

The  image  below  shows  the  visual  representation  for  a  late  arrival.  A  black  rectangle  on  the  timeline 
shows  when  the  UAV  is  going  to  arrive  at  the  labeled  target,  and  changes  from  black  to  red  when  the 
UAV  is  late.  This  can  also  be  seen  when  the  black  rectangle  moves  past  the  orange  window  of 
opportunity.  On  the  timeline  below,  the  red  rectangle  labeled  “T-1L”  indicates  that  the  UAV  is  going  to 
be  late  to  target  T-1L.  In  contrast,  the  black  rectangle  labeled  “T-2M”  indicates  that  the  UAV  is  going  to 
be  on  time  to  target  T-2M. 


L_ WP-1B |M 


1  TET-L, 

F 

■ 

T-2M 

T-1 L  T-2M  | 
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APPENDIX  B  SCENARIO  EVENTS 

These  slides  show  the  major  events  in  the  single  UAV  and  multiple  UAV  30  minute  test  scenarios. 


Timeline  Key 


■ 

■ 


Firing  Window  with  required  TOT 

•TOT  is  beginning  of  20  second  window 
Arming  Window 

•10  second  window  ending  at  required  TOT 
Deviation 

•Begins  at  time  listed  above  it  on  the  timeline 
Late  Arrival 

•Appears  at  time  listed  below  it  on  the  timeline 
Pop-up  Threat 

•Appears  at  time  listed  below  it  on  the  timeline  next  to  the  UAV 
loop  whose  timeline  it  is  on. 

Radio  Calls  to  be  acknowledged 

♦Red  when  in  conflict  with  a  firing  or  arming  window 

•Stack  on  each  other  if  within  a  few  second  interval 
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The  table  below  shows  side  by  side  images  of  the  timeline  display  and  map  display  for  the  single  UAV 
and  multi  UAV  scenarios.  The  displays  show  that  in  both  scenarios  the  operator  has  the  same  number  of 
tasks  to  complete.  The  only  difference  between  the  two  scenarios  is  that  for  the  multi  scenario  the  tasks 
are  divided  over  four  UAVs  instead  of  just  one  UAV. 


Single  UAV  Scenario 


Multi  UAV  Scenario 
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APPENDIX  C  PARTICIPANT  CONSENT  FORM 
CONSENT  TO  PARTICIPATE  IN 
NON-BIOMEDICAL  RESEARCH 

Developing  Decision  Support  for  Supervisory  Control  of  Multiple  Unmanned  Vehicles 

You  are  asked  to  participate  in  a  research  study  conducted  by  Professor  Mary  Cummings  Ph.D,  from  the 
Aeronautics  and  Astronautics  Department  at  the  Massachusetts  Institute  of  Technology  (M.I.T.).  You 
were  selected  as  a  possible  participant  in  this  study  because  the  expected  population  this  research  will 
influence  is  expected  to  contain  men  and  women  between  the  ages  of  18  and  50  with  an  interest  in  using 
computers.  You  should  read  the  information  below,  and  ask  questions  about  anything  you  do  not 
understand,  before  deciding  whether  or  not  to  participate. 

•  PARTICIPATION  AND  WITHDRAWAL 

Your  participation  in  this  study  is  completely  voluntary  and  you  are  free  to  choose  whether  to  be  in  it  or 
not.  If  you  choose  to  be  in  this  study,  you  may  subsequently  withdraw  from  it  at  any  time  without  penalty 
or  consequences  of  any  kind.  The  investigator  may  withdraw  you  from  this  research  if  circumstances 
arise  which  warrant  doing  so. 

•  PURPOSE  OF  THE  STUDY 

The  study  is  designed  to  evaluate  how  decision  support  tools  or  recommendations,  both  audio  and  visual, 
assist  an  operator  supervising  multiple  simultaneous  dynamic  tasks,  and  how  decision  support  assistance 
and  effectiveness  changes  as  workload  increases.  In  measuring  the  effectiveness  of  decision  support,  an 
operator’s  performance  and  situation  awareness  are  used  as  metrics.  Situation  awareness  is  generally 
defined  as  the  perception  of  the  elements  in  the  environment,  the  comprehension  of  the  current  situation, 
and  the  projection  of  future  status  of  the  related  system. 

•  PROCEDURES 

If  you  volunteer  to  participate  in  this  study,  we  would  ask  you  to  do  the  following  things: 

•  Attend  a  training  and  practice  session  to  learn  a  video  game-like  software  program  that  will  have 
you  supervising  and  interacting  with  multiple  unmanned  aerial  vehicles  (estimated  time  0.75 
hours). 

•  Practice  on  the  program  will  be  performed  until  an  adequate  level  of  performance  is  achieved, 
which  will  be  determined  by  your  demonstrating  basic  proficiency  in  monitoring  the  vehicles, 
redirecting  them  as  necessary,  executing  commands  such  as  firing  and  arming  of  payload  at 
appropriate  times,  using  decision  support  visualizations  and/or  recommendations  to  mitigate 
timeline  problems,  and  responding  to  radio  calls  by  clicking  an  acknowledge  button  on  the 
software  interface  (estimated  time  0.75  hours). 

•  Execute  two  thirty  minute  trials  consisting  of  the  same  tasks  as  above  (1  hour) 

•  Attend  a  debriefing  to  determine  your  subjective  responses  and  opinion  of  the  software  (10 
minutes). 

•  Testing  will  take  place  in  MIT  building  37,  room  301. 
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•  Total  time:  2-3  hours,  depending  on  skill  level. 

•  POTENTIAL  RISKS  AND  DISCOMFORTS 

There  are  no  anticipated  physical  or  psychological  risks  in  this  study. 

•  POTENTIAL  BENEFITS 

While  there  is  no  immediate  foreseeable  benefit  to  you  as  a  participant  in  this  study,  your  efforts  will 
provide  critical  insight  into  the  human  cognitive  capabilities  and  limitations  for  people  who  are  expected 
to  supervise  multiple  complex  tasks  at  once,  and  how  decision  support  tools  can  support  their  task 
management. 

•  PAYMENT  FOR  PARTICIPATION 

You  will  be  paid  $  10/hr  to  participate  in  this  study  which  will  be  paid  upon  completion  of  your  debrief. 
Should  you  elect  to  withdraw  in  the  middle  of  the  study,  you  will  be  compensated  for  the  hours  you  spent 
in  the  study. 

•  CONFIDENTIALITY 

Any  information  that  is  obtained  in  connection  with  this  study  and  that  can  be  identified  with  you  will 
remain  confidential  and  will  be  disclosed  only  with  your  permission  or  as  required  by  law.  You  will  be 
assigned  a  subject  number  which  will  be  used  on  all  related  documents  to  include  databases,  summaries  of 
results,  etc.  Only  one  master  list  of  subject  names  and  numbers  will  exist  that  will  remain  only  in  the 
custody  of  Professor  Cummings. 

•  IDENTIFICATION  OF  INVESTIGATORS 

If  you  have  any  questions  or  concerns  about  the  research,  please  feel  free  to  contact  the  Principal 
Investigator,  Mary  L.  Cummings,  at  (617)  252-1512,  e-mail,  missvc@mit.edu,  and  her  address  is  77 
Massachusetts  Avenue,  Room  33-305,  Cambridge,  MA  02139.  The  student  investigators  are  Hudson  D. 
Graham  (719-238-1713,  email:  hgraham@mit.edu),  and  Amy  Brzezinski  (617-276-6708, 
amybrz@MIT.EDU). 

•  EMERGENCY  CARE  AND  COMPENSATION  FOR  INJURY 

“In  the  unlikely  event  of  physical  injury  resulting  from  participation  in  this  research  you  may  receive 
medical  treatment  from  the  M.I.T.  Medical  Department,  including  emergency  treatment  and  follow-up 
care  as  needed.  Your  insurance  carrier  may  be  billed  for  the  cost  of  such  treatment.  M.I.T.  does  not 
provide  any  other  form  of  compensation  for  injury.  Moreover,  in  either  providing  or  making  such 
medical  care  available  it  does  not  imply  the  injury  is  the  fault  of  the  investigator.  Further  information  may 
be  obtained  by  calling  the  MIT  Insurance  and  Legal  Affairs  Office  at  1-617-253-2822.” 
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RIGHTS  OF  RESEARCH  SUBJECTS 


You  are  not  waiving  any  legal  claims,  rights  or  remedies  because  of  your  participation  in  this  research 
study.  If  you  feel  you  have  been  treated  unfairly,  or  you  have  questions  regarding  your  rights  as  a 
research  subject,  you  may  contact  the  Chairman  of  the  Committee  on  the  Use  of  Humans  as  Experimental 
Subjects,  M.I.T.,  Room  E25-143b,  77  Massachusetts  Ave,  Cambridge,  MA  02139,  phone  1-617-253- 
6787. 


SIGNATURE  OF  RESEARCH  SUBJECT  OR  LEGAL  REPRESENTATIVE 


I  understand  the  procedures  described  above.  My  questions  have  been  answered  to  my  satisfaction,  and  I 
agree  to  participate  in  this  study.  I  have  been  given  a  copy  of  this  form. 


Name  of  Subject 


Signature  of  Subject 


Date 


SIGNATURE  OF  INVESTIGATOR 


In  my  judgment  the  subject  is  voluntarily  and  knowingly  giving  informed  consent  and  possesses  the  legal 
capacity  to  give  informed  consent  to  participate  in  this  research  study. 


Signature  of  Investigator 


Date 
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APPENDIX  D  DEMOGRAPHICS  SURVEY 

MAUVE-MITUS  Demographic  Survey 


1.  Age: _ 

2.  Gender:  □  Male  □  Female 

3.  Occupation: _ 

If  student: 

a.  Class  Standing:  □  Undergraduate  □  Graduate 

b.  Major: _ 

If  currently  or  formerly  part  of  any  country’s  armed  forces: 

a.  Country /State: _ 

b.  Status:  □  Active  Duty  □  Reserve  □  Retired 

c.  Service:  □  Army  □  Navy  □  Air  Force  □  Other _ 

d.  Rank : _ 

e.  Years  of  Service: _ 

f.  Did  you  ever  serve  in  high  noise  environments?  □  Yes  □  No 

If  yes,  please  explain  what  the  duties  were,  how  long  the  shifts  were,  and  how  many  times  you 
served  these  shifts? 


4.  Do  you  have  experience  with  remotely  piloted  vehicles  (land,  sea,  air)? 

□  Yes 

□  No 

If  yes: 

a.  Vehicle  type( s )/class( es ): _ 

b.  Number  of  hours: _ 

5.  Do  you  have  experience  with  radios  such  as  those  used  for  communication  in  flying? 

□  Yes 

□  No 

If  yes: 

Please  explain: _ 


6.  Have  you  been  to  a  music  concert  in  the  last  Month? 

□  Yes 

□  No 

If  yes: 

a.  Concert  type: _ 

b.  When: _ 

7.  Do  you  have  any  hearing  loss? 

□  Yes 

□  No 
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If  yes: 

Please  explain: 


8.  How  often  do  you  play  video  games? 

□  Never 

□  Less  than  1  hour  per  week 

□  Between  1  and  4  hours  per  week 

□  Between  1  and  2  hours  per  day 

□  More  than  2  hours  per  day 

9.  Are  you  color  blind? 

□  Yes 

□  No 


If  yes: 

Which  type  of  color  blindness  (if  known). 
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APPENDIX  E  MAUVE-MITUS  TUTORIAL 


Spring  2007 

Q 

Multi- Aerial  Unmanned  Vehicle 
Experiment  (MAUVE) 
TUTORIAL 

1 

Hudson  Graham  -  MIT  Humans  and  Automation  Lab  |l|if 

Introduction 


Welcome! 

This  tutorial  is  designed  to  give  you  some  background  on  the  Multi- 
Aerial  Unmanned  Vehicle  Experiment  (MAUVE)  interface  before  you 
arrive  on  testing  day.  Please  take  the  time  to  look  over  the  following 
slides  and  come  prepared  with  questions.  Before  testing  you  will  be 
thoroughly  trained  on  the  actual  interface,  but  being  exposed  to  it 
beforehand  will  be  invaluable  in  speeding  up  this  process. 

Thank  you  in  advance  for  your  participation! 


Experiment  Overview 


In  this  experiment,  you  are  an  unmanned  aerial  vehicle  (UAV) 
operator  that  is  responsible  for  supervising  1  to  4  UAVs  collectively 
tasked  with  destroying  a  set  of  time-sensitive  targets  in  a 
suppression  of  enemy  air  defenses  mission.  The  area  contains  enemy 
threats  capable  of  firing  on  your  UAVs. 

The  UAVs  are  highly  autonomous,  and  therefore  only  require  high 
level  mission  execution  from  you.  The  UAVs  launch  with  a  pre¬ 
determined  mission  plan,  so  initial  target  assignments  and  routes 
have  already  been  completed  for  you.  Your  job  will  be  to  monitor 
their  progress,  re-plan  aspects  of  the  mission  in  reaction  to 
unexpected  events,  and  in  some  cases  manually  execute  mission 
critical  actions  such  as  arming  and  firing  of  payloads. 

The  interface  we  have  developed  for  this  experiment  is  called  the 
Multi-Aerial  Unmanned  Vehicle  Experiment  (MAUVE)  and  will  be 
referred  to  by  this  name  from  here  out. 


Objectives 

Your  primary  objective  in  this  mission  is: 

To  make  sure  the  UAV(s)  maximize  the  number  of  targets  engaged  as  well  as 
arriving  back  to  the  base  safely. 

Supervision  of  the  UAVs  can  be  broken  down  into  the  following 
prioritized  sub-tasks,  from  highest  priority  to  lowest: 

1.  Return  to  base  (RTB)  within  the  time  limit  for  the  mission  (this  limit  will  be 
clearly  marked). 

2.  Comply  with  recovery  rules  for  course  deviations. 

3.  Comply  with  recovery  rules  for  target  late  arrivals. 

4.  Destroy  all  targets  before  their  time  on  target  (TOT)  window  ends. 

5.  Avoid  taking  damage  from  enemies  by  avoiding  all  threat  areas. 

6.  Acknowledge  all  "Push"  radio  calls. 

These  sets  of  objectives  will  often  conflict  with  one  another.  In  these 
cases,  you  must  perform  the  actions  that  have  the  highest  priority 
first. 

Your  performance  will  be  judged  based  on  how  well  you  follow  the  above 
priorities. 
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Audio  Alerts 


To  help  you  meet  your  objectives  you  will  receive  auditory  signals  for  both 
course  deviations  and  late  arrivals.  Both  are  induced  by  unanticipated  high 
winds  along  the  planned  flight  path. 

Course  deviations  are  when  a  UAV  is  blown  off  of  the  planned  path.  It  is 

significantly  deviated  when  you  visually  see  that  the  UAV  has  left  the  course  line. 
Deviations  may  occur  over  targets  as  well. 


Late  arrivals  are  when  the  UAV  has  hit  stronger  than  anticipated  head  winds  and 
slows  down.  As  a  result  it  will  now  be  late  to  the  next  target. 

Your  test  proctor  will  provide  further  training  as  to  what  these  auditory 
signals  sound  like  during  the  test  day  training. 

Other  auditory  sounds  to  be  familiar  with  are:  (Note  all  three  are  the  same 
because  all  are  related  to  a  new  message  in  your  message  box.) 

€  «®. 

For  new  messages  in  For  pop-up  threats  For  when  your  UAV  is 

your  message  boxes  being  fired  upon 

while  flying  through 
a  threat  area. 


Color  Coding 


Throughout  the  displays  you're  about  to  see,  the  following  color  coding  is  used 
to  indicate  each  of  the  5  possible  actions  a  UAV  can  perform  in  MAUVE: 


UAV  Action 

Color 

Enroute 

Gray 

Loitering 

Blue 

Arming  Payload 

Yellow 

Firing  Payload 

Orange 

Return  to  Base 

Green 

Displays  -  Overview 


During  the  experiment,  you  will  see  two  side-by-side  displays  that  contain 
the  following  major  elements: 

•  Left  Display 

-  Mission  Time 

-  Map  Display 

-  Mission  Execution 

•  Right  Display 

-  Unmanned  Aerial  Vehicle  (UAV)  Status 

-  Decision  Support 

-  Chat  Box 

-  UAV  Health  &  Status  Updates 

The  following  slides  will  show  these  displays  in  detail  and  explain  how  to  use 
them  properly. 


Left  Display  -  Overview 


The  three 
major  screen 
elements  on 
the  left  display 
are: 


1 

Mission 

Execution 

2 

Mission 

Time 
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Right  Display  -  Overview 


Left  Display  -  Detail 

The  following  slides  detail  all  of  the  elements  contained  on  the  left 
display,  in  this  order: 

•  Map  Display 

•  Mission  Execution 

•  Mission  Time 


Key  Map  Display  Elements 

1.  Active  Target 

2.  Waypoint 

3.  Base 

4.  UAV 

5.  Threat  Area 


Mission  Plans 

•  The  solid  black  lines  indicate 
each  UAV's  current  mission  plan 

•  The  currently  selected  mission 
plan  is  highlighted  green 

•  One  UAV  will  always  be 
highlighted  with  a  red  border. 
This  UAV  has  the  greatest 
course  deviation.  It  corresponds 
to  the  course  deviation  auditory 
signal  you  will  be  receiving. 


Map  Display  -  Detail  -  1 
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Map  Display  -  Detail  -  2 


Naming  Conventions 

•  UAVs 

-  Numbered  1-4 

•  Targets 

-  T-XXP  where  XX  =  target  number  and  P  =  priority 

-  Priority  may  be  High  (H),  Medium  (M),  or  Low  (L) 

-  Examples: 

■  T-1H  -  Target  1  a  high  priority  target 

■  T-12M  -  Target  12  a  medium  priority  target 

■  T-23L  -  Target  23  a  low  priority  target 

•  Waypoints  (WP) 

-  WP-XY  where  X  =  UAV#  the  waypoint  is  associated  with  and  Y  =  waypoint 
letter 

-  Examples:  WP-1A,  WP-2C 

•  Threats/ Hazards 

-  H-XXX  where  XXX  =  threat  number 

-  Example:  H-001,  H-012 
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Mission  Time  -  Detail 


The  mission  time  display  element  shows  the  following: 

1.  Absolute  Time 

-  These  clocks  are  on  the  left  side  of  the  "Mission  Time"  title 

-  Two  formats 

■  Current  Time:  Current  mission  time 

■  End  Time:  End  time  of  the  current  mission 

2.  Relative  Time 

-  These  clocks  are  on  the  right  side  of  the  "Mission  Time"  title 

-  Two  formats 

■  Elapsed  Time:  How  long  has  elapsed  since  the  start  of  the  scenario 

■  Remaining  Time:  How  long  remains  until  the  mission  is  over 


Example  Mission  Time  Display  Element 


Current: 

End: 

MISSION 

Elapsed: 

Remaining: 

|  12:02:43  Z  | 

|  12:30:00  Z  | 

|  TIME 

|  00:02:43  | 

\  00:27:17  | 

Mission  Execution  -  Detail  -  2 

Mission  Execution  Functions 


UAV  4 


1.  Arm  Payload 

-  This  button  is  only  enabled  if  the  UAV 
is  selected  while  directly  on  top  of  a 
target,  and  within  the  arming  or 
firing  windows 

2.  Fire  Payload 

-  This  button  is  only  enabled  if  the  UAV 
is  selected  while  directly  on  top  of  a 
target,  armed,  and  within  the  firing 
window  for  that  particular  target 

3.  Skip  Target 

-  This  button  is  used  if  you  decide  to 
skip  a  target  because  you  are  going 
to  be  late  to  it.  It  causes  the  UAV  to 
skip  the  next  target/waypoint  and 
move  to  the  next  waypoint/target 


0 
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Mission  Execution  -  Detail  -  1 


Each  UAV  has  its  own  mission  execution  bar. 

To  bring  it  up  on  the  left  display,  click  anywhere  on  the  desired  UAV's  status 
window  on  the  right  display  OR  on  the  UAV  icon  itself  on  the  left  display. 

Light  green  highlighting  around  the  UAV's  status  bar  and  its  current  mission 
plan  on  the  map  display  tell  you  which  UAV/route  is  currently  selected 


In  the  display  below  UAV  4  is  highlighted  so  this  is  the  mission  execution 
bar  on  the  leftside  of  the  left  display. 
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Health  &  Status  Updates 
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Mission  Execution  -  Detail  -  3 


Mission  Execution  Functions 

4.  Reset  Navigation 

-  Click  anytime  you  have  a  significant 
course  deviation  and  want  to  return  to 
the  planned  course;  it  will  return  the 
UAV  to  the  next  plotted 
waypoint/target. 

-  Causes  the  UAV  to  be  inactive  while 
resetting;  you  may  not  be  able  to  arm 
or  fire  with  that  UAV  when  its 
navigation  system  is  resetting. 

5.  Radio  Monitoring 

-  Radio  chat  from  the  Boston  ground 
control  will  be  playing  and  you  must 
click  the  "Acknowledge  PUSH"  each 
time  "push"  is  called  on  the  radio  chat. 
Hint:  Push  calls  usually  come  in  pairs 
(but  not  always);  the  tower  and  an 
aircrew  or  vice  versa. 

6.  Target  Assignment  Queue 

-  You  will  not  be  using  this  portion  of  the 
display. 


UAV  4 


0 

0 

0 


Right  Display  -  Detail 


The  following  slides  detail  all  of  the  elements  contained  on  the  right 
display,  in  this  order: 

•  UAV  Status 

•  Health  &  Status  Updates 

•  Decision  Support 

•  Chat  Box 


UAV  Status  -  Detail  -  1 


The  UAV  status  display  shows  the  following 
real-time  information  for  each  UAV: 

•  Status  /  Current  Action 

-This  is  written  out  as  well  as 
represented  by  the  color  of  the  UAV 
icon  to  the  right 

-For  example:  a  blue  UAV  would  be 
loitering.  This  means  it  has  arrived 
over  the  target  but  is  there  before  the 
arming  and  firing  window  so  is  in  a 
holding  pattern  waiting  for  these 
windows. 

•  Current  Target  Name 

•  Position  in  Latitude  &  Longitude 

-You  will  not  need  this  in  the  scenario. 

-Given  in  degrees,  minutes,  and  seconds 

•  Altitude 

-This  is  a  static  number  that  is  not  used 
in  the  scenario  in  any  way 


Example  UAV  Status  Display  Element 


Status:  Returning  to  base 

Current  Target: 

Latitude:  26* *1 2' 18  N 
Longitude:  46*55' 55.  E 

Altitude:  1 0000  MSL  ^ 
Course:  57* 

Speed:  200  kts  1 

Payload  Ready:  No  1  1 

Status:  Enroute 

Current  Target:  T-5H 
Latitude:  21  *09'  1  3  N 
Longitude:  45*55' 1 7.  E 

Altitude:  IOOOOMSL  ^ 
Course:  65- 

Speed:  200  kts  2 

Payload  Ready:  No 

Status:  Armed 

Current  Target:  T-3H 
Latitude:  21*30' 35.  N 
Longitude:  41  *44' 45.  E 

Altitude:  1 0000  MSL  ^ 
Course:  38*  f  \ 

Speed:  200  kts  3 

Payload  Ready:  Yes  1  1 

Status:  Loitering  at  target 
Current  Target:  T-4M 
Latitude:  28*57' 38.  N 
Longitude:  49*25' 45  E 

Altitude:  10000  MSL  ^ 
Course: -138- 
Speed:  200  kts  4 

Payload  Ready:  No 

(continued  on  next  slide) 


UAV  Status  -  Detail  -  2 


The  UAV  status  display  shows  the  following  real¬ 
time  information  for  each  UAV: 


Course 


Example  UA  V  Status  Display  Element 


-You  will  not  need  this  during  the  scenario,  status:  Returning  to  base  Attitude:  ioooomsl 

Current  Tarnet!  Course:  57* 

-0°  indicates  due  north;  increases  in  a 
clockwise  manner 


Current  Target: 
Latitude:  26*1 2'  18.  N 
Longitude:  46*55'  55  E 


Course:  57- 
Speed:  200  kts 
Payload  Ready:  No 


a 


•  Speed 

-The  UAVs  are  set  to  travel  at  a  constant 
speed  of  200  kts. 

-If  there  is  significant  head  wind  the  UAV 
may  slow  down.  This  can  precipitate  late 
target  arrivals 

•  Payload  Ready 


Status:  Enroute  Altitude:  1 0000  MSL 

Current  Target:  T-5H  Course:  65* 

Latitude:  21*09'  1.3  N  Speed:  200  kts  | 

Longitude:  45*55'  1 7  E  Payload  Ready:  No  1 

(h 

Status:  Armed  Altitude:  1 0000  MSL 

Current  Target:  T-3H  Course:  38- 

Latitude:  21  *30  35  N  Speed:  200  kls  i 

Longitude:  41*44' 45.  E  Payload  Ready:  Yes  1 

-This  reflects  whether  the  UAV  has  a 
payload  ready  for  the  current  target 

-Will  say  "Yes"  if  the  UAV  is  armed,  "No"  if 
not 


Status:  Loitering  at  target 

Current  Target:  T-4M 
Latitude:  28‘57'  38.  N 
Longitude:  49*25' 45.  E 


Altitude:  1 0000  MSL 
Course: -138- 
Speed:  200  kts 
Payload  Ready:  No 


UAV  Health  &  Status  Updates  -  Detail 


The  Health  &  Status  Updates  box  contains  messages  from  specific  UAVs  intended  to 
inform  the  operator.  Messages  are  color  coded  as  follows: 

•  Red  =  UAV  Health  messages 

-  UAV  is  under  fire  from  a  threat 

-  Again,  a  standard  audio  alert  will  play  when  you  receive  red  messages. 

•  Bold  Black  =  UAV  Status  messages,  action  required 

-  UAV  is  available  to  arm  or  fire 

•  Black  =  UAV  Status  messages,  no  action  required 

-  UAV  has  completed  arming  or  firing 


Example  Health  &  Status  Updates  Window 

Health  &  Status  Updates 

UAV  1  (12:09:02)  ->  Under  Fire  from  threat  H-001 
UAV  1  (12:09:22)  ->  Under  Fire  from  threat  H-001 
UAV  1  (12:09:42)  ->  Under  Fire  from  threat  H-001 
UAV  4  (12:13:39)  ->  Available  to  arm  for  target  T-4M 
UAV  4  (12:13:41)  ->  Arming  for  T-4M 
UAV  4  (12:13:45)  ->  Arming  completed  for  target  T-4M 
UAV  4  (12:13:49)  =>  Available  to  fire  for  target  T-4M 


Decision  Support  -  Detail  -  1 


The  active  level  of  decision  support  contains  a  visual  representation  of  what  and 
when  targets  are  approaching  through  a  relative  timeline  and  projective 
decision  support  display  for  each  UAV. 

•  The  arming  and  firing  windows  cannot  by  changed  solely  at  the  will  of  the 
the  operator,  i.e.  Operators  may  request  time  on  target  (TOT)  delay 
requests,  but  must  get  approval  before  the  arming  and  firing  window  will 
be  moved  back  (if  approved). 


Example  Active  Decision  Support  Window 


Decision  Support  -  Remember  the  Color  Coding 


Color  coding  is  an  important  element  of  the  decision  support,  so  take  a  look  at 
it  again! 


UAV  Action 

Color 

Enroute 

Gray 

Loitering 

Blue 

Arming  Payload 

Yellow 

Firing  Payload 

Orange 

Return  to  Base 

Green 

Decision  Support  -  Detail  -  2 


Arming  and  firing  elements  are  color  coded  in  the  same  way  as  corresponding 
UAV  actions.  For  each  target  the  following  information  is  represented: 

1.  Arming  Window  =  Yellow 

-  10  seconds  long  and  takes  approximately  3-7  seconds  to  arm. 

-  Payload  for  the  relevant  target  may  be  armed,  but  not  fired  during 
this  time 

-  Always  occurs  immediately  before  the  firing  window 

2.  TOT/ Firing  Window  =  Orange 

-  20  seconds  long  and  takes  approximately  3-7  seconds  to  fire. 

-  Payload  must  be  fired  at  the  relevant  target  during  this  time 

-  In  addition  to  the  arming  window,  a  payload  may  also  be  armed 
during  this  time 

-  Target  name  is  printed  vertically  in  the  center  of  the  window, 
priority  is  printed  above  the  window. 


T-1L  T-2M 


Decision  Support  -  Detail  -  3 


Mission  planning  information  reflects  when  UAVs  will  reach  important  points 
in  the  scenario,  such  as: 

1.  Waypoints/ Loiterpoints/ Base  =  Black  Triangles 

-  Names  are  printed  above  the  relevant  UAV's  timeline 

2.  UAV  Arrival  at  Targets  =  Black  Rectangles 

-  Names  are  printed  below  the  relevant  UAV's  timeline 

-  Note  that  each  target  name  will  appear  twice  on  the  timeline, 
once  for  when  the  UAV  will  arrive  at  that  target  and  once  at  the 
center  of  that  target's  firing  window 

3.  Late  UAV  Arrival  =  Red  Rectangles 

-  Black  Rectangle  turns  red  and  moves  past  the  target  to  when  the 
UAV  will  arrive. 
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Decision  Support  -  Detail  -  5 


The  decision  support  visualization  (DSV)  helps  the  user  manage  the 
schedule  by  showing  timeline  issues  and  projecting  "what  if" 
conditions  of  the  effects  on  the  timeline  based  upon  user 
decisions. 

•  Each  UAV's  DSV  is  uses  emergent  features  to  show  problems 
that  currently  exist  or  that  may  exist  if  a  TOT  delay  is  given. 

•  No  issues  (late  target  arrivals)  are  indicated  by  no  rectangles 
being  displayed. 

-  The  picture  below  shows  the  DSV  for  when  there  is  no  late 
arrivals.  It  depicts  this  by  having  no  rectangles  above  the 
line  on  the  left  side  of  the  display  in  the  "Late  Arrival" 
section. 

Note:  The  DSV  will  be  inactive  except  for  when  you  are  going  to  be 
late  to  a  medium  or  high  priority  target. 


T-3H 

- 

Req.  Delay 

Probability: 

- 

I  NONE  | 

Projection 


Decision  Support  -  Detail  -  4 


The  active  level  of  decision  support  aids  the  user  by  identifying  possible  late 
target  arrivals  and  areas  of  potential  high  workload  on  the  timeline. 

•  A  late  target  arrival  is  defined  as  when  a  UAV  will  arrive  to  a  target  after 
its  scheduled  time  on  target 

•  Corrective  actions  for  late  arrivals  are  based  on  the  priority  of  the  target 
you  are  projected  to  be  late  to.  For: 

-  Low  priority  targets,  skip  them  by  clicking  "Skip  Target"  on  the 
navigation  screen. 

-  Medium  priority  targets,  either  skip  them  or  use  the  decision  support 
visualization  (DSV)  to  possibly  request  a  delay.  Remember  your 
priorities  of  wanting  to  hit  all  the  targets.  See  the  next  four  slides  for 
an  explanation  of  how  to  use  the  DSV  (shown  below). 

-  High  priority  targets,  use  the  DSV  and  then  decide  whether  to  request 
a  TOT  delay  or  to  skip  the  target  by  clicking  "Skip  Target" 

Note:  The  corrective  actions  above  should  only  be  taken  when  a  late  arrival  is 
projected  by  the  red  rectangle  on  the  screen  and  your  audio. 


Laie 

I 


Projection 


T-6M 

- 

T-7H 

T-3H 

T-13L 

Decision  Support  -  Detail  -  6 


•  Late  arrivals  are  represented  by  a  rectangle  occurring  on  the 
DSV  above  the  line  on  the  left  side  of  the  display  in  the  "Late 
Arrivals"  section.  It  will  also  be  highlighted  yellow  as  it  is 
below. 

-  The  target's  priority  is  indicated  within  the  rectangle  and 
by  the  rectangle's  size.  The  higher  the  priority  of  the 
target  that  the  UAV  will  be  late  for  the  taller  the  rectangle 
will  be. 


Late  Arrivals 


H- 


T-ll 

T-2M 


Projection 


Req,  Delay 


Prota  ability: 

NONE 


Decision  Support  -  Detail  -  7 


•  Below  the  center  line  is  for  the  "what  if"  condition;  after  the 
user  selects  a  target  that  they  might  request  a  TOT  delay  for  it 
will  show  the  projected  future  late  arrivals  for  that  UAV  below 
the  centerline  on  the  left  side  of  the  display  in  the  "Projection" 
section. 


-  The  example  below  shows  that  if  a  TOT  delay  request  is 
granted  for  target  T-16H,  the  UAV  will  then  be  late  to  a 
Low  Priority  target. 

-  The  Probability  in  the  bottom  right  of  this  display  shows 
the  likelihood  of  a  TOT  delay  being  granted.  The  further  in 
advance  a  delay  is  requested  the  higher  the  likelihood  of  it 
being  granted.  Do  not  request  a  delay  again  if  your  first 
request  for  a  delay  on  that  target  is  denied. 


Late  Arrivals 

B— 

Projection 


Chat  Box  -  Detail 


The  chat  box  contains  a  time  history  of  all  human  interactions.  These  interactions  are 
color  coded  as  follows: 


•  Red  =  Intelligence  updates 

-  Again,  a  standard  audio  alert  will  play  when  you  receive  red  messages. 

•  Black  =  Message  to/from  Base,  no  response  required 

-  Messages  that  inform  you,  but  do  not  require  a  specific  response 

The  chat  box  is  purely  informational.  It  will  provide  you  with  updates,  but  you  will  not 
input  anything  in  the  chat  box. 


Example  Message  History  Window 

Menage  History 

Intelligence  (12:04:01)  =>  Reporting  the  following  emerging  threat:H-006 
(12:05:23)  ->  Reset  Navigation:  UAV  3 
(12:06:52)  =>  PUSH  Acknowledged 
(12:06:56)  =>  PUSH  Acknowledged 

Base  (12:08:24)  ->  Request  to  move  the  TOT  for  T-3M  was  approved 
Base  (12:08:24)  =>  New  TOT  for  T-3M  is  12:12:60Z  -  12:13:20Z 


Decision  Support  -  Detail  -  8 


Each  UAV  possesses  a  DSV  display  to  help  the 
user  understand  the  potential  effects  of 
decisions 

•  A  list  of  all  the  mission  targets  on  the  timeline 
appears  to  the  right  of  each  UAV's  DSV  display. 

•  In  using  the  display  to  the  right  for  the  top  UAV: 

-  The  user  is  considering  requesting  a  delay 
for  T-7H,  a  high  priority  target.  However, 
it  shows  they  will  now  be  late  to  another 
high  priority  target  even  with  this  delay. 
This  is  where  you,  as  the  user,  will  have  to 
make  a  value  call.  Do  you  request  a  delay 
or  skip  the  target  by  clicking  "Skip  Target?" 
Most  of  the  time  you  will  not  want  to 
request  a  delay  if  you  know  it  is  going  to 
create  another  delay. 

•  In  using  the  display  to  the  right  for  the  second 
UAV: 


-  The  user  is  considering  requesting  a  delay 
for  T-16H,  a  high  priority  target.  This  will 
result  in  the  UAV  being  late  to  a  low 
priority  target.  So  in  this  case  you,  as  the 
user,  would  want  to  request  this  delay 
because  it  means  you  can  hit  the  high 
priority  target  for  the  trade-off  of  now 
missing  a  low  priority  target. 


h 

Projection 


Projection 


T-6M 

Req.  Delay 

T-7H 

T-3H 

T-13L  - 

Probability: 

-  MED 

T-16H 

Req.  Delay 

Probability: 

-  MED 

T-14L 

T-11M 

T-8L 

Late  Arrivals 


Projection 


T-2M 

~  Req.  Delay 

T-12L 

T-4H 

Probability: 

T-10H 

|  NONE  | 

Late  Arrivals 


Projection 


T-15M 

T-1L 

Req.  Delay 

T-5H 

T-9L 

Probability: 

1 - 1 

▼ 

| NONE  | 

Conclusion 


You  are  now  ready  to  proceed  to  hands-on  training  with  the  MAUVE 
interface.  Remember  to  bring  any  questions  you  have  to  the 
experimenter  on  testing  day! 


APPENDIX  F  POST-EXPERIMENT  SURVEY 


1.  How  did  the  audio  cues  help  or  hinder  you  in  managing  [ate  arrivals ? 


2.  How  did  the  audio  cues  help  or  hinder  you  in  managing  course  deviations ? 


3.  How  would  you  change  the  audio  cues  for  the  [ate  arrivals  to  have  them  better  aid 
you  during  your  mission? 


4.  How  would  you  change  the  audio  cues  for  the  course  deviations  to  have  them  better 
aid  you  during  your  mission? 
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5.  In  general  how  would  you  change  all  the  audio  to  help  you  with  the  control  during 
your  mission? 


6.  Please  express  any  other  comments  you  may  like  to  share: 
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APPENDIX  G  GLM  ANALYSIS:  SPSS  OUTPUT 


Course  Deviation  Reaction  Time  (for  4  audio  conditions) 

Tests  of  Normality 


Kolmogorov-Smirnov(a) 

Shapiro-Wilk 

Audio_Scheme 

Statistic 

Df 

Sig. 

Statistic 

Df 

Sig. 

DSing 

Combination 

.175 

10 

.200(*) 

.949 

10 

.653 

Continuous  Late 
Arrival 

.192 

10 

.200(*) 

.912 

10 

.296 

Continuous 

Course 

Deviation 

.180 

10 

.200(*) 

.893 

10 

.181 

Threshold 

.113 

9 

.200(*) 

.978 

9 

.952 

D _ Multi 

Combination 

.193 

10 

.200(*) 

.927 

10 

.415 

Continuous  Late 
Arrival 

.192 

10 

.200(*) 

.918 

10 

.339 

Continuous 

Course 

.267 

10 

.042 

.843 

10 

.048 

Deviation 

Threshold 

.165 

9 

.200(*) 

.964 

9 

.836 

*  This  is  a  lower  bound  of  the  true  significance, 
a  Lilliefors  Significance  Correction 


Levene's  Test  of  Equality  of  Error  Variances(a) 


F 

df  1 

df2 

Sig. 

DSing 

3 

35 

D _ Multi 

3 

35 

Tests  the  null  hypothesis  that  the  error  variance  of  the  dependent  variable  is  equal  across  groups, 
a  Design:  lntercept+Audio_Scheme  Within  Subjects  Design:  scenario 

Tests  of  Within-Subjects  Contrasts 


Measure:  MEASURE1 


Source 

scenario 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

Sig. 

scenario 

Linear 

14.029 

1 

14.029 

3.215 

.082 

scenario  * 
Audio_Schem 

Linear 

10.780 

3 

3.593 

.824 

.490 

e 

Error(scenari 

o) 

Linear 

152.711 

35 

4.363 
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Tests  of  Between-Subjects  Effects 


Measure:  MEASURE1 
Transformed  Variable:  Average 


Source 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

Sig. 

Intercept 

8998.235 

1 

8998.235 

1089.907 

.000 

Audio_Scheme 

71.289 

3 

23.763 

2.878 

.050 

Error 

288.959 

35 

8.256 

Tukey  Comparison 

Multiple  Comparisons 


Measure:  MEASURE1 
Tukey  HSD 


Mean 

Difference 

95%  Confidence  Interval 

(1)  Audio Scheme 

(J)  Audio Scheme 

(i-j) 

Std.  Error 

Sig. 

Lower  Bound 

Upper  Bound 

Combination 

Continuous  Late  Arrival 

-1.6292 

.90862 

.294 

-4.0796 

.8213 

Continuous  Course 
Deviation 

.3667 

.90862 

.977 

-2.0838 

2.8171 

Threshold 

-1.7949 

.93352 

.237 

-4.3125 

.7227 

Continuous  Late  Arrival 

Combination 

1 .6292 

.90862 

.294 

-.8213 

4.0796 

Continuous  Course 
Deviation 

1 .9958 

.90862 

.144 

-.4546 

4.4463 

Threshold 

-.1657 

.93352 

.998 

-2.6834 

2.3519 

Continuous  Course 
Deviation 

Combination 

-.3667 

.90862 

.977 

-2.8171 

2.0838 

Continuous  Late  Arrival 

-1.9958 

.90862 

.144 

-4.4463 

.4546 

Threshold 

-2.1616 

.93352 

.114 

-4.6792 

.3560 

Threshold 

Combination 

1 .7949 

.93352 

.237 

-.7227 

4.3125 

Continuous  Late  Arrival 

.1657 

.93352 

.998 

-2.3519 

2.6834 

Continuous  Course 
Deviation 

2.1616 

.93352 

.114 

-.3560 

4.6792 

Based  on  observed  means. 


D 

Min  Max 

q* 

P 

Sing 

D1 

Comb-Mod 

-2.02 

-4.48 

0.45 

-3.55 

D2 

Comb-Osc 

-0.46 

-2.92 

2.00 

-0.81 

D3 

Comb-Thresh 

-2.73 

-5.20 

-0.27 

-4.81 

0.02 

D4 

Mod-Osc 

1.56 

-0.90 

4.02 

2.74 

D5 

Mod-Thresh 

-0.72 

-3.18 

1.75 

-1.26 

D6 

Osc-Thresh 

-2.28 

-4.74 

0.19 

-4.00 

Multi 

D7 

Comb-Mod 

-1.24 

-3.70 

1.22 

-2.18 

D8 

Comb-Osc 

1.19 

-1.27 

3.65 

2.09 

D9 

Comb-Thresh 

-0.86 

-3.32 

1.61 

-1.50 

D10 

Mod-Osc 

2.43 

-0.03 

4.90 

4.28 

Dll 

Mod-Thresh 

0.39 

-2.08 

2.85 

0.68 

D12 

Osc-Thresh 

-2.05 

-4.51 

0.42 

-3.60 
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Course  Deviation  Reaction  Time  (for  2  audio  alerts) 


Tests  of  Normality 


Audio_Scheme 

_Osc 

Kolmogorov-Smirnov(a) 

Shapiro-Wilk 

Statistic 

Df 

Sig. 

Statistic 

df 

Sig. 

DSing 

Oscillating  Alert 

.145 

20 

.2000 

.933 

20 

.179 

Beep  Alert 

.130 

19 

.2000 

.971 

19 

.799 

D _ Multi 

Oscillating  Alert 

.180 

20 

.089 

.940 

20 

.237 

Beep  Alert 

.107 

19 

.2000 

.967 

19 

.718 

*  This  is  a  lower  bound  of  the  true  significance, 
a  Lilliefors  Significance  Correction 


Levene's  Test  of  Equality  of  Error  Variances(a) 


F 

df  1 

df2 

Sig. 

D_Sing 

.328 

1 

37 

.570 

D _ Multi 

.167 

1 

37 

.685 

Tests  the  null  hypothesis  that  the  error  variance  of  the  dependent  variable  is  equal  across  groups, 
a  Design:  lntercept+Audio_Scheme_Osc  Within  Subjects  Design:  scenario 

Tests  of  Within-Subjects  Contrasts 


Measure:  MEASURE1 


Source 

scenario 

Type  III  Sum 
of  Squares 

Df 

Mean  Square 

F 

Sig. 

Scenario 

Linear 

14.534 

1 

14.534 

3.311 

.077 

scenario  * 

Linear 

Audio_Scheme 

1.089 

1 

1.089 

.248 

.621 

_Osc 

Error(scenario) 

Linear 

162.403 

37 

4.389 

Tests  of  Between-Subjects  Effects 


Measure:  MEASURE1 
Transformed  Variable:  Average 


Source 

Type  III  Sum 
of  Squares 

Df 

Mean  Square 

F 

Sig. 

Intercept 

9007.397 

1 

9007.397 

1146.991 

.000 

Audio_Scheme_Osc 

69.684 

1 

69.684 

8.874 

.005 

Error 

290.564 

37 

7.853 
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Transformed  (Natural  Log)  Late  Arrival  Reaction  Time 


Tests  of  Normality 


Kolmogorov-Smirnov(a) 

Shapiro-Wilk 

Audio_Scheme 

Statistic 

Df 

Sig. 

Statistic 

df 

Sig. 

LN_L_Sing 

Combination 

.162 

10 

.200(*) 

.958 

10 

.768 

Continuous  Late 
Arrival 

.205 

10 

.200(*) 

.943 

10 

.583 

Continuous 

Course  Deviation 

.246 

10 

.088 

.839 

10 

.043 

Threshold 

.180 

9 

.200H 

.915 

9 

.349 

LN_L_Multi 

Combination 

.156 

10 

.200(*) 

.962 

10 

.812 

Continuous  Late 
Arrival 

.215 

10 

.200(*) 

.903 

10 

.234 

Continuous 

Course  Deviation 

.242 

10 

.098 

.915 

10 

.319 

Threshold 

.170 

9 

.200(*) 

.977 

9 

.944 

*  This  is  a  lower  bound  of  the  true  significance, 
a  Lilliefors  Significance  Correction 


Levene's  Test  of  Equality  of  Error  Variances(a) 


F 

df  1 

Df2 

Sig. 

LN_L_Sing 

.651 

3 

35 

.587 

LN_L_Multi 

1.583 

3 

35 

.211 

Tests  the  null  hypothesis  that  the  error  variance  of  the  dependent  variable  is  equal  across  groups, 
a  Design:  lntercept+Audio_Scheme  Within  Subjects  Design:  scenario 

Tests  of  Within-Subjects  Contrasts 


Measure:  MEASURE1 


Source 

scenario 

Type  III  Sum 
of  Squares 

Df 

Mean  Square 

F 

Sig. 

Scenario 

Linear 

6.513 

1 

6.513 

20.737 

.000 

scenario  * 
Audio_Schem 

Linear 

1.619 

3 

.540 

1.719 

.181 

e 

Error(scenari 

o) 

Linear 

10.992 

35 

.314 

Tests  of  Between-Subjects  Effects 


Measure:  MEASURE1 
Transformed  Variable:  Average 


Source 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

Sig. 

Intercept 

244.837 

1 

244.837 

305.851 

.000 

Audio_Scheme 

8.028 

3 

2.676 

3.343 

.030 

Error 

28.018 

35 

.801 
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Tukey  Comparison 


Multiple  Comparisons 


Measure:  MEASURE1 
Tukey  HSD 


Mean 

Difference 

95%  Confidence  Interval 

(1)  Audio Scheme 

(J)  Audio Scheme 

(i-j) 

Std.  Error 

Sig. 

Lower  Bound 

Upper  Bound 

Combination 

Continuous  Late  Arrival 

.0092 

.28293 

1.000 

-.7539 

.7722 

Continuous  Course 
Deviation 

-.7099 

.28293 

.076 

-1.4729 

.0532 

Threshold 

.0633 

.29069 

.996 

-.7206 

.8473 

Continuous  Late  Arrival 

Combination 

-.0092 

.28293 

1.000 

-.7722 

.7539 

Continuous  Course 
Deviation 

-.7191 

.28293 

.071 

-1.4821 

.0440 

Threshold 

.0542 

.29069 

.998 

-.7298 

.8381 

Continuous  Course 
Deviation 

Combination 

.7099 

.28293 

.076 

-.0532 

1 .4729 

Continuous  Late  Arrival 

.7191 

.28293 

.071 

-.0440 

1.4821 

Threshold 

.7732 

.29069 

.054 

-.0107 

1 .5572 

Threshold 

Combination 

-.0633 

.29069 

.996 

-.8473 

.7206 

Continuous  Late  Arrival 

-.0542 

.29069 

.998 

-.8381 

.7298 

Continuous  Course 
Deviation 

-.7732 

.29069 

.054 

-1.5572 

.0107 

Based  on  observed  means. 


D 

Min  Max 

q* 

P 

Sing 

D1 

Comb-Mod 

-0.22 

-0.95 

0.52 

-1.28 

D2 

Comb-Osc 

-0.60 

-1.34 

0.13 

-3.57 

D3 

Comb-Thresh 

-0.16 

-0.89 

0.57 

-0.95 

D4 

Mod-Osc 

-0.39 

-1.12 

0.34 

-2.29 

D5 

Mod-Thresh 

0.06 

-0.68 

0.79 

0.33 

D6 

Osc-Thresh 

0.44 

-0.29 

1.18 

2.62 

Multi 

D7 

Comb-Mod 

0.24 

-0.50 

0.97 

1.39 

D8 

Comb-Osc 

-0.82 

-1.55 

-0.08 

-4.82 

0.02 

D9 

Comb-Thresh 

0.29 

-0.44 

1.02 

1.70 

D10 

Mod-Osc 

-1.05 

-1.78 

-0.32 

-6.21 

0.06 

Dll 

Mod-Thresh 

0.05 

-0.68 

0.78 

0.31 

D12 

Osc-Thresh 

1.10 

0.37 

1.84 

6.53 

0.07 

Sing-Multi 

Min  Max 

Q* 

P 

Comb 

Mod 

Osc 

Thresh 

-0.75088 

-1.48 

-0.02 

-4.44 

0.04 

-0.29899 

-1.03 

0.43 

-1.77 

-0.96236 

-1.69 

-0.23 

-5.69 

0.03 

-0.30183 

-1.03 

0.43 

-1.79 
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Transformed  (Natural  Log)  Late  Arrival  Reaction  Time  (with 
Combination/Continuous  Late  Arrival/Threshold  Combined  against  Continuous 
Course  Deviation) 


Tests  of  Normality 


Kolmogorov-Smirnov(a) 

Shapiro-Wilk 

Late_Comparison 

Statistic 

Df 

Sig. 

Statistic 

df 

Sig. 

LN_L_Sing 

Continuous  Course 
Deviation 

.246 

10 

.088 

.839 

10 

.043 

Comb_ContLate_T 

hresh 

.090 

29 

.200(*) 

.959 

29 

.316 

LN_L_Multi 

Continuous  Course 
Deviation 

.242 

10 

.098 

.915 

10 

.319 

Comb_ContLate_T 

hresh 

.149 

29 

.101 

.961 

29 

.341 

*  This  is  a  lower  bound  of  the  true  significance, 
a  Lilliefors  Significance  Correction 


Levene's  Test  of  Equality  of  Error  Variances(a) 


F 

df  1 

Df2 

Sig. 

LN_L_Sing 

1.055 

1 

37 

.311 

LNLMulti 

3.216 

1 

37 

.081 

Tests  the  null  hypothesis  that  the  error  variance  of  the  dependent  variable  is  equal  across  groups, 
a  Design:  lntercept+Late_Comparison  Within  Subjects  Design:  scenario 

Tests  of  Within-Subjects  Contrasts 


Measure:  MEASURE1 


Source 

scenario 

Type  III  Sum 
of  Squares 

Df 

Mean  Square 

F 

Sig. 

scenario 

Linear 

7.476 

1 

7.476 

23.730 

.000 

scenario  * 
Late_Compari 

Linear 

.954 

1 

.954 

3.029 

.090 

son 

Error(scenario 

) 

Linear 

11.657 

37 

.315 

Tests  of  Between-Subjects  Effects 


Measure:  MEASURE1 
Transformed  Variable:  Average 


Source 

Type  III  Sum 
of  Squares 

Df 

Mean  Square 

F 

Sig. 

Intercept 

228.000 

1 

228.000 

300.624 

.000 

Late_Comparison 

7.984 

1 

7.984 

10.528 

.002 

Error 

28.062 

37 

.758 

43 


NASA  TLX  Scores 


Tests  of  Normality 


Audio_Schem 

e 

Kolmogorov-Smirnov(a) 

Shapiro-Wilk  | 

Statistic 

Df 

Sig. 

Statistic 

Df 

Sig. 

TLX_Sing 

Combination 

.136 

10 

.2000 

.969 

10 

.882 

Modulated 

.175 

10 

.2000 

.952 

10 

.688 

Oscillating 

.209 

10 

.2000 

.886 

10 

.153 

Threshold 

.231 

9 

.181 

.939 

9 

.574 

TLXJVIulti 

Combination 

.256 

10 

.063 

.846 

10 

.051 

Modulated 

.146 

10 

.2000 

.963 

10 

.818 

Oscillating 

.151 

10 

.2000 

.922 

10 

.376 

Threshold 

.129 

9 

.2000 

.957 

9 

.769 

*  This  is  a  lower  bound  of  the  true  significance, 
a  Lilliefors  Significance  Correction 


Levene's  Test  of  Equality  of  Error  Variances(a) 


F 

df  1 

df2 

Sig. 

TLX_Sing 

2.424 

3 

35 

.082 

TLXJVIulti 

1.038 

3 

35 

.388 

Tests  the  null  hypothesis  that  the  error  variance  of  the  dependent  variable  is  equal  across  groups, 
a  Design:  lntercept+Audio_Scheme  Within  Subjects  Design:  scenario 

Tests  of  Within-Subjects  Contrasts 


Measure:  MEASURE1 


Source 

Scenario 

Type  III  Sum 
of  Squares 

Df 

Mean  Square 

F 

Sig. 

scenario 

Linear 

3.364 

1 

3.364 

.058 

.811 

scenario  * 

Linear 

Audio_Schem 

215.197 

3 

71.732 

1.234 

.312 

e 

Error(scenari 

Linear 

2035.166 

35 

58.148 

o) 

Tests  of  Between-Subjects  Effects 


Measure:  MEASURE1 
Transformed  Variable:  Average 


Source 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

Sig. 

Intercept 

96127.322 

1 

96127.322 

188.194 

.000 

Audio_Scheme 

1244.548 

3 

414.849 

.812 

.496 

Error 

17877.551 

35 

510.787 
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Combination  Continuous  Late  Continuous  Threshold 
Arrival  Course 

Deviation 


Audio_Scheme 


■  Single  UAV 
0  Mult  UAVi 


Missed  Radio  Calls 


Tests  of  Normality 


Kolmogorov-Smirnov(a) 

Shapiro-Wilk 

Audio Scheme 

Statistic 

Df 

Sig. 

Statistic 

df 

Sig. 

Missed_Sing 

Combination 

.234 

10 

.127 

.893 

10 

.184 

Continuous  Late 
Arrival 

.165 

10 

.200H 

.919 

10 

.351 

Continuous  Course 
Deviation 

.245 

10 

.089 

.865 

10 

.087 

Threshold 

.221 

9 

.200(*) 

.910 

9 

.315 

Missed_Multi 

Combination 

.214 

10 

.200(*) 

.814 

10 

.022 

Continuous  Late 
Arrival 

.249 

10 

.079 

.894 

10 

.190 

Continuous  Course 
Deviation 

.177 

10 

.200(*) 

.909 

10 

.273 

Threshold 

.217 

9 

.200(*) 

.914 

9 

.349 

*  This  is  a  lower  bound  of  the  true  significance, 
a  Lilliefors  Significance  Correction 


Levene's  Test  of  Equality  of  Error  Variances(a) 


F 

df  1 

df2 

Sig. 

Missed_Sing 

2.150 

3 

35 

.111 

Missed_Multi 

1.684 

3 

35 

.188 

Tests  the  null  hypothesis  that  the  error  variance  of  the  dependent  variable  is  equal  across  groups, 
a  Design:  lntercept+Audio_Scheme  Within  Subjects  Design:  scenario 
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Tests  of  Within-Subjects  Contrasts 


Measure:  MEASURE1 


Type  III  Sum 

Source 

scenario 

of  Squares 

df 

Mean  Square 

F 

Sig. 

scenario 

Linear 

34.307 

1 

34.307 

1.411 

.243 

scenario  * 

Linear 

Audio_Schem 

65.774 

3 

21.925 

.902 

.450 

e 

Error(scenari 

o) 

Linear 

850.944 

35 

24.313 

Tests  of  Between-Subjects  Effects 


Measure:  MEASURE1 
Transformed  Variable:  Average 


Source 

Type  III  Sum 
of  Squares 

df 

Mean  Square 

F 

Sig. 

Intercept 

7966.465 

1 

7966.465 

92.043 

.000 

Audio_Scheme 

210.033 

3 

70.011 

.809 

.498 

Error 

3029.300 

35 

86.551 
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